Measuring Impact - but how, who and what?

In the framework of the CAKE-CoSeC EDI working group, six of the working group members formed a focus group with the title “Measuring Impact”, or, more specifically “Best practices for measuring impact of EDIA initiatives”. We set out with ambitious plans, aiming to develop concrete actions, identify key definitions to share and provide recommendations for institutions to improve their approaches to measuring and collecting information on the impact of their EDIA activities. We did not only plan to curate KPIs which can lead to change, but also analyse those, create an “evidence-based best practice guide”, “proactive solutions” and liaise with other related groups to “co-create output […] for embedded reach into UKRI policy”.

Looking back through our notes of those first meetings they show an admirable enthusiasm - and slight naivety. This might come as a surprise, given that most of us have worked in, or contributed to, the EDI space for years. But, to see it positively, it shows that there is still this spark, this drive to change things for the better, and we have not all been disillusioned completely by the reality of workplaces and academia.

Our meetings quickly became lengthy discussions. What do we even mean by “Impact”, and how do we measure it? Are we talking about qualitative or quantitative measures? Are case studies just anecdotal evidence, or useful examples of impact? What about negative examples, are they worthwhile sharing? Who is this about, do we focus on researchers and digital Research Technical Professionals (dRTPs), or just one of these groups? How are dRTPs actually defined? And who is this work for, who are the stakeholders? How does all of this work in small and in big institutions? And so on and so on.

Our lofty goals from the start necessarily needed to be boiled down, not alone because of the time commitment - as so often, enthusiasm for the topic was not matched by time allowance from everyone’s day job. We decided that we would shift from planning to test out things ourselves to writing up our thoughts and developing recommendations, which subsequently can be picked up by anyone - ourselves, a subset of our working group, or a different group.

Let’s start then by contributing our thoughts about measuring impact to the wider discussion:

“You improve what you measure” is a frequently heard statement (see, e.g., success of health tracking devices and running watches), and certainly also true for EDIA initiatives. However, you cannot measure everything (easily). Reverting to case studies and anecdotal evidence, on the other hand, is not sufficient to make real change - key metrics are often missing/unknown, or the data to quantify them is not present. For example, tracking numbers of minoritised staff members is important to see where an institution is lacking.
Additionally, there are metrics which can be used as proxies, or “canaries in the gold mine”, for EDI-supportive environments and psychological safety in the workplace, such as physical and mental health, the uptake of support offers, or the number of grievances/complaints.

An undisputed challenge for measuring EDI KPIs in some areas is the high-level of under-reporting because of stigma and fear of discrimination. For example, people may be reluctant to disclose some protected characteristics, even in anonymous surveys, for a variety of reasons.

Importantly, the management, measurement and responsibility for EDI KPIs should not be owned solely by one person in the team “wearing the EDI hat”. We need accountability at all levels across organisations - in particular, members of senior leadership in an institution need to be evaluated against EDI success. Only then will real action be taken beyond checkbox exercises.

Our working group will continue working towards developing a list of KPIs and recommendations for the Digital Research Infrastructure community within the UK. What this will look like in the end we do not know - but we hope that it will be an increment on the current state of EDI, and will be taken up, discussed and continued by the next enthusiastic group coming together and believing that change is possible.