ENRESSH Newsletters

N° 3 April 2018

Welcome

Professor Ioana Galleron

Dear reader,

If you were a participant to the last ENRESSH meeting in Lisbon, you could have heard several times “we couldn’t do this before”. “This” is gathering information about the evaluation systems of SSH research in virtually all European countries and sketching a typology. “This” is 53 interviews with recent PhD holders from 16 countries, offering unique insights about perceptions on scientific quality and career strategies of young European SSH researchers. “This” is a training school on achieving impact, as well as an analysis of publication patterns in the SSH. Many other reports, papers, presentations are in preparation, such as a platform for gathering internal peer-reviews on books in publishing houses.

At the end of its second grant period, ENRESSH is progressing full steam towards the realisation of its objectives, and this newsletter tries to reflect this. Do not hesitate to look at our website for more, or even to drop us a line!

Wishing you a pleasant reading,

Ioana Galleron

Chair of ENRESSH

People

CollaborationENRESSH

NBABE: Non-Bibliometric Aspects of Book Evaluation

Geoffrey Williams (Université de Bretagne-Sud) and Jon Holm (The Research Council of Norway) initiated the study Non-Bibliometric Aspects of Book Evaluation (NBABE).

Books, and book chapters, are often seen as a defining feature of SSH output. The answer is more complex and essential questions have to be asked about the nature of these publications and their importance to SSH research.

These questions were utmost in the minds of Geoffrey and Jon since they first met at a HERA board meeting. It was however the round table on books at RESSH 2016 that really crystallised the need for a closer analysis within ENRESSH. Bibliometric aspects have been a feature of WG3 work on databases from the outset as books, and chapters, do pose a problem. NBABE seeks to make this issue a transversal one that takes into account the how and why SSH researchers use this form of dissemination and how they can be evaluated.

Many questions arose from the RESSH debate. Some of these are aired at a first meeting in Helsinki and a second one in Lisbon. Amongst these were how to classify different types of book publication, what is academic and what not and whether databases should restrict themselves to a definition of academic, how and whether books and chapters undergo a real peer-review system, especially as in a collective work a reviewer many be competent in only one aspect and not the entirety, whether the onus of quality control can be shifted to publishers via a label system. Above all, NBABE will need to talk to researchers and understand their strategy rather than purely the strategies laid down from above.

NBABE is just one aspect of the overall task of enquiring into book evaluation. Much has been done, much is being done the task is to draw the threads together. That is what networking and ENRESSH are about.

New Members

UCD Information and Communication Studies – Dr Lai Ma

Introducing Lai Ma

I am Assistant Professor at School of Information and Communication Studies at University College Dublin, Ireland. My work is mainly concerned with the interrelationship between epistemology, information infrastructure (primarily bibliographic and citation databases), and its cultural and social affordances and implications. As a humanist in information science, I feel deeply about the need to articulate the importance of theoretical and conceptual work — even when we are thinking about big data, technology and science, as critical thinking is essential to consider ethical issues, as well as potential cultural and societal implications and consequences. Coming from information science, I also hope that my knowledge in bibliometrics, scholarly communication, and metadata would be contributive to projects undertaken in ENRESSH.

Why I joined ENRESSH? First, I met some of the members in other conferences and really enjoyed the vivid and inspiring conversations about SSH research (and yes, they are fun to be with too)! Second, I am enthusiastic in projects that highlight the significance of SSH research not only in scholarly discourse, but also the society at large, as the value of SSH research has been undermined in many research evaluation systems. Third, members of ENRESSH come from a wide range of backgrounds, which creates a good platform for the exchange of ideas and creating fruitful and impactful projects to inform policy-making.

I am very happy to be part of ENRESSH and enjoyed my first meeting in Lisbon in March. The working groups are focused and productive, people are friendly and communicative. I look forward to working with many of you!

Other new members

Dr. Filippo Nereo, principal lecturer, Coventry University, United Kingdom

Agne Girkontaite, doctoral student, Vilnius University, Lithuania

Joshua Eykens , doctoral student, Ghent University, Belgium

Dr. Tony Ross Hellauer, senior post-doctoral researcher, Know-center, Austria

Message to ENRESSH

Dear ENRESSH Members,

As the Chair of the Humanities in the European Research Area (HERA) Network Board, I am happy to express our warm sympathy towards your work. HERA is a “network of national funding agencies committed to leading and developing funding opportunities for humanities researchers in Europe”. Over the years, it became the central instrument for funding the humanities at the transnational level in Europe. Hera has always been committed to not only designing new research programmes, but also to coordinating national research policies in the humanities and defining methods for evaluating the impact of humanities research.

Is arts and humanities scholarship pointless? Do they have social or economic benefits? Can they be measured? Should they be measured? Does such measurement demonstrate they are worth funding? The question of ‘what is the public value of arts and humanities research?’ has become a fundamental concern for societies and funders across the world. Especially now, in times of diminishing research budgets across much of the world, and when the basic pillars of the future of European research funding are being set up.

As scholars, of course we know that the humanities are incredibly important. Yet, a lot of high quality research and knowledge exchange work remains somehow invisible. There is also a widespread suspicion of the media and the way in which humanities research is represented in everyday life. Furthermore, there is reluctance to engage in current affairs and debates that could potentially be informed by project findings. Finally, there is the frustration of not being able to reach politicians and policy makers through research activities. Therefore, clever dissemination strategies, knowledge exchange and demonstrable impact are becoming vital for the existence and further development of humanities scholarship. That is why your Action is so important for all members of the social sciences and humanities community.

From our point of view your work is really essential. It provides a valuable knowledge and feedback we could share among our national partners. They all are facing more or less similar problems while trying to evaluate research in their own countries. Additionally, it also seems to be crucial for a general visibility of the field in scope of the European agenda. We wish you luck with your endeavour and we look forward to your results.

Dr. Wojciech Sowa

Chair of the HERA Network Board

News

Evaluation Frameworks

Dr. Alexander Hasgall

How to evaluate SSH research in its diversity?

Disciplines have their own epistemologies and practices, different universities have different missions. How can evaluation procedures reflect such a rich diversity? ENRESSH-member Alexander Hasgall is first author of a report by swissuniversities. Swissuniversities is the rectors’ conference for all universities, universities of applied science and universities of teacher education in Switzerland. The report summarizes a research program on research evaluation in the SSH. Also, it presents ten theses for an appropriate evaluation of research that takes into account the rich diversity of research at universities. The report includes contributions based on research by fellow ENRESSH’ers Sven Hug, Michael Ochsner and Karin Byland.

The ten theses are as follows:

1. Disparate scientific cultures between and also within disciplines, require different forms of evaluation
2. Evaluation procedures reflect, and take into consideration, different strategic profiles between research entities and institutions
3. When designing and implementing evaluation procedures, a bottom-up approach ensures pertinent and adequate results
4. Making the reason and the underlying norms behind an evaluation is part of the evaluation process
5. Clearly defining the form and goals of evaluations
6. Quantitative measurements may be made in addition to qualitative evaluations, but cannot replace them
7. Different dimensions in research impacts
8. Adapting evaluations to differing contexts
9. Research impact lasts over time and is not limited to published works
10. Meeting scientific requirements

We strongly encourage you to read the report and have a closer look at the detailed formulation of the theses starting on page 39.

Relevance and Impact

It’s been a very busy quarter for us in Working Group 2 (Societal Impact). The most important event for us has been the Training School in Croatia; a high-level think tank bringing together experts from the Netherlands, UK, and Norway with an enthusiastic trainee group. The week saw participants reflect at length on the wicked issues of the theories, practices and policies underpinning the evaluation of humanities societal impact. One of the working groups in the training school is currently working on the approach they developed for measuring societal impact into a publication, and one of the trainers requested hard copies of the assignments completed by trainees – this shows in a nutshell the depth and profundity of deliberations, which provide a solid foundation for the working group as we move into the second half of ENRESSH.

We also came together in Lisbon to share our recent experiences and to plan our future work. We received fascinating presentations from our STSMs:

• Stefan de Jong (the University of Manchester) pointed out that whatever bottlenecks there were in SSH impact in south and east Europe, it was not for a want of effort from their academics.
• Agne Girkontaite (Vilnius University) argued that creating impact involves tensions between the worlds of science and society, and that early career researchers needed guiding in how to profit from and exploit these complementarities rather than freezing in the face of them.
• Eirikur Smari Sigurdarson (University of Iceland) argued that one way to better understand humanities’ impact was through the epistemic capabilities and societal capacities they mobilised, his findings from the first part of his STSM this spring in Valencia.

In 2018 we will focus on two issues. Firstly is local impact contexts that researchers face – if SSH researchers are trying their best to create impact in south and east Europe, then we want to understand what it is about these local contexts that is holding them back. Secondly is to understand the tensions that SSH early career researchers face in trying to make these contradictory ‘worlds’ complementary (together with the ESR SIG). We are currently designing concrete research plans for these activities for presentation in Ljubljana. I hope to see you there, and if you have any thoughts or suggestions on these two issues, then please contact me!

(Paul Benneworth, University of Twente).

Databases and Metrics

In a comparison of SSH publication patterns across eight countries we show that the use of language and publication types are influenced by each country’s cultural and historic heritage.

Through a short survey we welcome all comments to this paper!

We collected data on all peer-reviewed SSH publications and specifically for the disciplines of economics and business, law, and philosophy and theology for the Czech Republic, Denmark, Finland, Flanders (Belgium), Norway, Poland, Slovakia, and Slovenia. No previous study has brought together data from such a wide range of comprehensive coverage national bibliographic databases.

With data on 272,376 peer-reviewed publications from the period of 2011–2014 analyzed, the study provides unique insight into the difference in publication patterns between and within fields. We observe that the publication patterns are stable and quite similar in West European and Nordic countries, whereas in Central and Eastern European countries the publication patterns demonstrate considerable changes over time. In all countries, the share of journal articles and the share of publications in English is on the rise.

We welcome your comments on the observations made in this paper, which is freely available online here. Please leave your comments via this short survey or send them directly via email to Emanuel Kulczycki. We intend to discuss all comments in an online forum soon.

On behalf of all co-authors,

Emanuel Kulczycki, Tim Engels and Janne Pölönen

The World of SSH: General News

Position paper on SSH in FP9

Eleven universities across Europe call for a strong position of the social sciences and humanities in FP9. Their call is supported by more than 60 other universities and organisations, including ENRESSH, and more than 400 individuals.

The position paper includes five recommendations:

• “The creation of an SSH Platform, analogous to ETPs, can consolidate this collaboration and structure the way this community provides input to scoping papers, work programmes, societal challenges and missions. The platform would also facilitate the search for SSH evaluators.

The support for multidisciplinary collaborations can be increased, by

• encouraging researchers to think about how to generate academic and societal impact when drafting proposals,
• recognising that all disciplines crucially contribute to the goal of an inclusive and prosperous Europe and
• investing in high-quality SSH research and innovation opportunities.
• A target budget for SSH spending of 10% of the overall civil research budget for FP9 will help in better facilitating the highest-quality SSH activities and consequently activating its full potential to academic and societal impact for the future of Europe.”

Conference on impact 

The Impact of Research and Innovation Policy at the Crossroads of Policy Design, Implementation and Evaluation conference takes place at 5-6 November, 2018 in Vienna, Austria.

It engages around 250 participants from all over Europe and international partner countries. It will gather academics, evaluators, research managers, R&I policy makers, R&I councils and funding agencies to debate the challenges of generating, understanding and assessing societal impact through R&I policy and the implications for evaluation theory and practice.

Thematic areas addressed by the conference are:

1) The nature of impact-oriented R&I policy

2) Design and implementation of impact-oriented R&I policies

3) Leading edge concepts, tools and methods to assess impact of R&I policy

4) Effects of and policy learning from impact evaluation

The submission deadline for papers is May 15, 2018.

Publications

Glossary

Rita Faria is assistant professor in criminology at the University of Porto. Stefan de Jong is post-doctoral researcher in science studies at The University of Manchester. Both joined the Training School on impact in the social sciences and humanities in Zagreb.

Stefan: ‘What was it like in your sub group? The exercise in evaluating those Norwegian impact case studies?’

Rita: ‘We were so critical! Nothing convinced us of any impact. So interesting, as earlier today we discussed how difficult it is to describe impacts.’

Stefan: ‘Indeed, we also turned in blood-thirsty animals…’

Rita: ‘It’s interesting to see what being handed power to judge does to people. In that sense, it reminded me of the Stanford prison experiment. Give a group of reasonable people power over another group and within no time it gets completely out of hand.’

Stefan: ‘Yeah, that’s a good one! And our victims were not even in the same room. What is the name of that other experiment again? The one where the victim is in another room and where a figure of authority encourages a person to inflict more pain, even when the victim is screaming aloud?’

Rita: ‘Are you referring to the Milgram experiment?’

Stefan: ‘That’s the one. Peer review in impact assessment is like the Stanford experiment with a dash of Milgram.’

Rita: ‘We should be more aware of that when we are reviewing, that in such a situation we have power over another person. And that we should use that power sensibly and responsibly.’

Stefan: ‘And it is known that especially in the social sciences and humanities were are pretty harsh to one another, for example in reviewing scientific papers or grant applications, due to our critical nature. Meanwhile, in the natural sciences they are more inclined to see the positive and exciting aspects of each other’s ideas, they are more appreciative apparently…’

Next Keyword

The next keyword is forthcoming

Calendar

Events

16 – 18 April

Annual conference of the European Association of Research Managers and Administrators

Brussels, Belgium

29th of May – 1st of June 2018

SARIMA Annual conference

Johanesburg, South Africa

4-7 June 2018

Conference of the International Network of Research Management Societies

Edinburgh, UK

6-8 June 2018

European Forum for Studies of Policies for Research and Innovation

Paris, France

25-28 July 2018

European Association for the Study of Science and Technology

Lancaster, UK

5-8 September 2018

Triple Helix Conference

Manchester, UK

Abstract deadline March 30!

12-14 September 2018

23rd International Conference on Science and Technology Indicators

Leiden, Netherlands

5-6 November 2018

Impact of Research and Innovation Policy at the Crossroads of Policy Design, Implementation and Evaluation

Vienna, Austria