The post Reflections on the REF 2021 appeared first on Digital Science.

UoA Classification for the Research Excellence Framework

We developed a REF Unit of Assessment classification system to enable institutions to assess the quality of their research. The system has been implemented using automated allocation of UoA codes (based on REF 2021) to documents on the Dimensions platform based on category definitions defined by machine learning. Using the system, institutions are able to compare their institutionally held data in Dimensions with a world view on their staff outputs in a REF context to retrieve a fuller picture of its output portfolio, including OA status and accompanying bibliometrics.

The system has the capability to give an institutional view on publications and grants and a Unit of Assessment view in a REF timeframe. The system enables institutions to compare their output with other institutions, providing the ability to see UOA classified research output on an ongoing basis rather than seeing it as a snapshot every REF period, thereby providing further and deeper institution-wide insights.

The post UoA classification for the REF appeared first on Digital Science.

Juergen Wastl is Director of Academic Relations and Consultancy at Digital Science…

Briony Fane is a Research Analyst at Digital Science…

Bo Alroe has worked with research management and administration since 2004, and currently as Director of Strategy with Digital Science…

Introduction

In this analysis, we have used Dimensions on Google BigQuery to analyse the Impact Case Studies from REF 2014…

Of the 6,975 Impact Case Studies submitted to REF 2014, 6,737 were included in the publicly available HEFCE dataset…

Using Google BigQuery to connect external datasets to Dimensions to gain important insights

Thanks to HEFCE’s (now Research England) freely available database of REF research submissions…

Prevalence of SDG-related research underpinning Impact Case Studies

Alignment of SDGs within the Dimensions categorisation and REF-related ‘types of impact’

SDGs can be clustered into three overarching pillars: Environmental, Economic and Social…

Comparing citation averages for SDG-related underpinning research versus non SDG-related underpinning research

REF Impact Case Studies were categorised into eight types by HEFCE…

Looking ahead, we might expect that research underpinning Impact Case Studies in REF 2021…

• Main Panel C (social sciences) contributes most and is evident in all SDGs…
• Main Panel A contributes predominantly through SDG3 Health and Well-Being…
• Main Panel B mostly contributes to SDG7 and SDG13…
• Main Panel D accommodates a more diverse portfolio…

Conclusion

This blog has revealed a starting point for us looking at the scope for interpreting SDGs through the lens of Impact…

Footnotes

1: The document can be viewed here

2: Each underpinning research publication may be associated with more than one SDG classification…

3: The SDG ‘wedding cake’, developed at the Stockholm Resilience Centre…

The post In search of SDGs in REF impact case studies appeared first on Digital Science.

The results of the next REF exercise will again be used to inform the allocation of government funding to universities for research, almost £2 billion of annual research investment. The previous exercise, in 2014, assessed nearly 200,000 research outputs and 7,000 impact case studies submitted by more than 52,000 academics in 154 institutions. This event will help support universities and research institutions in delivering world-class research and ensure they adhere to the developing assessment criteria.

Digital Science Consultant Data Scientist, Martin Szomszor, will be speaking at 15:55 on Thursday, 14 September 2017. His talk is titled, REF 2021: Game Changing?

Where: Manchester Conference Centre

Time: 08:30 – 16:30

Research Impact: Strengthening the Excellence Framework will explore the next steps and rules for the Research Excellence Framework in 2021, offering practical guidance on delivering high-quality submissions across disciplines, institutions and projects.

You can expect to hear about:

• Learn about the future direction for research funding and assessment as the government reviews of governance, structure and evaluation develop.
• Practical advice and solutions to assist universities in improving the quality of REF submissions and interdisciplinary research for 2021.
• Explore new models and future iterations of research performance and evaluation.
• The impact of the Research Excellence Framework on organisational culture and how the assessment process will inform future submissions and assessment.
• Learn how to best evidence impact, excellence and diversity across a variety of disciplines and research projects.
• Examples of good practice and improvements for the next REF exercise.
• Insight into the intended and unintended consequences of REF and how the next exercise will differ.
• The future role of metrics and data in evidencing research impact and processes.
• Assistance in developing the quality of research outputs and demonstrating originality, significance and rigour in applications.
• How to evidence impact beyond academia in terms of reach and significance.
• Practical ways of developing research strategies, support for staff and students, and delivering research infrastructures and collaborations.
• How to benchmark outcomes from Research Excellence Framework assessments towards 2021.
• Discover how new models of research, funding and collaboration can support excellence.
• Take advantage of knowledge sharing, professional networking and contribute to wider thinking about higher education and research.

REGISTRATION CLOSED

The post Research Impact: Strengthening the Excellence Framework appeared first on Digital Science.

‘Publication patterns in research underpinning impact in REF 2014’ is the first report to analyse all research publications submitted for research assessment between 1988 and 2014. The study found that at least 42 per cent of the research cited in impact case studies in the 2014 Research Excellence Framework (REF2014) had been submitted as research outputs to the various prior research assessments. This suggests that research of scholarly significance leads to societal impact, but also emphasises the broad base of research from which impact stems.

The analysis covers almost 1 million submitted research outputs and more than 35,000 impact case study references across 25 years and five assessment exercises. The report considers the relationship between submitted outputs (as evidence of academic research quality) and impact case study references (as evidence of societal and economic impact). It explores what the data says about the selection of research outputs for assessment, and about changes in selection behaviour as a response to assessment.

Key findings include:

• Over time, journal articles have become a more commonly submitted output type (when compared with books, conference proceedings and other media). This is despite the fact that HEFCE analysis comparing quality profile by output type found that monographs were awarded the highest proportion of four star scores (Note 1).
• In early research assessment cycles, the research outputs submitted tended to be those published towards the end of the publication period. In Science and Engineering recent exercises saw submissions from earlier in the cycle in all relevant disciplines for all institutions.
• Of the impact case study references with digital object identifiers, 42 per cent were submitted to Research Assessment Exercises or the 2014 Research Excellence Framework (REF2014) as research outputs; these are evenly spread between publication years from 1996 to 2013.
• Earliest references are skewed towards recent publications for social science and humanities. In science and engineering, the date of the earliest reference in each impact case study is uniformly spread through the eligible publication period.
• The overlap between case study references and submitted outputs for assessment appears to be higher in more applied areas, and lower in the core humanities and pure sciences.
• For science and engineering, the case study references have highest overlap with the research outputs submitted in early cycles. The overlap between case study references and assessment outputs submitted to REF2014 is higher in social sciences.

Quote from Steven Hill, Head of research policy at HEFCE:

‘The innovative analysis in this report gives us, for the first time, evidence that research of all types leads to wider benefits for the economy, society and the environment. The findings emphasise the importance of supporting a diverse range of research to deliver impact.’

Quote from Dr Jonathan Adams, Chief Scientist & Head of Consultancy at Digital Science:

‘This huge body of work provides us with uniquely rich data on research activity, submitted outputs and case studies across 25 years and five assessment exercises. This is one of several perspectives that Digital Science has created on these data. The data will be available for the research base to mine in order to assess, uncover and understand the scholarly significance of the UK research base.’

About Digital Science

Digital Science is a technology company serving the needs of scientific and research communities at key points along the full cycle of research. It invests in and incubates research software companies that simplify the research cycle, making more time for discovery. The consulting group delivers data-driven analytical reports and technology solutions to research policy and management clients such as higher education institutions, charities and funding agencies, publishers and policy bodies. It is operated by global media company, the Holtzbrinck Publishing Group. Visit https://www.digital-science.com and follow @digitalsci on Twitter.

Download The Report

Media contact

David Ellis, Press, PR & Social Manager, Digital Science: Mobile +61 447 783 023, d.ellis@digital-science.com

The post Report shows investing in academic excellence does lead to research impact appeared first on Digital Science.

The UK’s most recent Research Excellence Framework (REF2014), for the first time, evidenced the economic and societal impact of university research through a total of 6,975 case studies. This Digital Research Report on ‘The Diversity of UK Research and Knowledge’ introduces this dataset and contains visualisations of the knowledge networks underpinning the impact of UK university research.

REF2014 Case Study Similarity Network

This interactive visualisation of the REF2014 impact case studies is a key exhibit in the Digital Research Report. Each case study is a node in the network. The links between case studies represent the similarity of text describing the background research. You can zoom in to explore clusters of related content.

Click on individual case study nodes to reveal case study titles and links to the full text. The menu in the top right corner allows selection of individual REF subject area panels to which the case studies were submitted. Colours indicate which panel they are:

• Green: Biomedical sciences
• Red: physical sciences and engineering
• Blue: social sciences
• Yellow: arts and humanities

The post The Diversity Of UK Research And Knowledge – A Digital Science Research Report appeared first on Digital Science.

The UK’s Research Excellence Framework (REF2014) produced a fantastic new database of evidence about the impact of university research. In the past we have had data on research funding, training and publications. Now we can add to that the impact of research on the economy, society, the environment, health, education and across the arts.

REF2014 included a total of 6,975 case studies capturing the work of 50,000 researchers working in 154 institutions and grouped into 36 disciplinary units of assessment (UOAs) in four overarching subject panels: life sciences; engineering and physical sciences; social sciences; and arts and humanities. Each four page study included the underpinning research, details of the impact, and corroborative materials.

In 2014, the Higher Education Funding Council for England (HEFCE, which runs the REF) commissioned Digital Science to turn these case studies into a searchable database and website. We worked with the Policy Institute at King’s College London to analyse the contents and King’s produced a great overview report delving into that.

It’s a lot of data. How do you work out what’s there? Each case study focusses on priority areas in science, policy and society – but what areas? How closely are they related? Digital Science started with a detailed text analysis and then compared pairs of documents: on the website you can find click to ‘View similar case studies’.

That’s fine, but what does the network of similarity look like? The online version (accessible here) shows you the knowledge clusters, it identifies the main subject panels of of the REF that the work is drawn from, and it also allows you to go straight to each case study in a cluster to look at the research. This makes it very easy to get both the big picture and the detail that it contains.

The UOA categories are too broad to tell you much detail about the research underlying the case studies, so we also looked at the text from a different angle to see which specific Fields of Research (FoRs) the researchers were drawing on. We assigned up to three FoRs to a case study because we found that most case studies crossed disciplinary boundaries. Impactful research is usually also interdisciplinary research, sometimes on a grand scale.

The case studies are linked by FoRs they have in common, this creates a second network which we can visualise and which allows people to quickly see how research fields link to one another. The way in which the fields link up in our visualisation looks remarkably like the ultimate key to human innovation – the brain! So we have called these networks Digital Science BrainScans, and we can create a BrainScan for each university and college that makes up the underlying database.

Click on the image for the interactive visualisation:

The post Visualising Research Impact appeared first on Digital Science.

Digital Science has completed the collation of more than 6,500 case study documents into the REF Impact Case Study database.  It is the first time universities have attempted to describe and quantify the far-reaching impact of their work.

The case studies demonstrate the impact of the academic research carried out by 154 of the UK’s universities and higher education institutions over the past twenty years. The freely available website and its underpinning database contains a wealth of information about UK research and its impact on health, society, the economy and the environment, in a format that is easy to search and categorise.

Digital Science, working in conjunction with the Policy Institute at King’s College London and Nature Publishing Group was commissioned by the Higher Education Funding Council for England (HEFCE) on behalf of the four UK funding bodies[1], Research Councils UK and the Wellcome Trust to build the online database and analyse the impact case studies submitted to the 2014 Research Excellence Framework (REF).

The Policy Institute at Kings produced an analytical synthetic report  of the case studies.  Nature Publishing Group has published a supplement that highlights a dozen selected case studies at the leading edge of impact, demonstrating the socio-economic and policy impact of UK research and illustrating its global reach.  The report will be printed in Nature on March 26^th and will be free to access online here.

The REF Impact Case Study database allows a new range of comparisons between institutions and against national reference profiles and benchmarks via indexed data that enables the exploration and grouping of the case studies to show the positive impact that UK research has made to the world.  It provides a broad canvas of some of the highest impact of excellent research achievements in science, arts and humanities disciplines. It is the first time universities have attempted to describe and quantify the far-reaching impact of their work.

The Digital Science team processed each REF impact case study to normalise the text into an easily readable format and to add index data to help explore and group the content. The analysis also tagged universities’ specialist research areas and impact focus, providing an invaluable tool to examine where research funding has underpinned impact, and for companies to see where they might best collaborate with universities in their R&D projects.

Using geotagging methods applied to the text, the Digital Science methodology also enables users to gain insights into the impact in other countries or – for the UK – within cities and regions.

Digital Science has provided links from research referenced in the case studies to both traditional publication databases such as Thomson Reuters’ Web of Science, for which the company provided augmented links, and leading alternative metrics provider Altmetric, to show article level metrics, where applicable.  Altmetric collates the data showing the attention scholarly research gathers from social media, newspapers, government policy documents and other sources for mentions of scholarly articles.

Jonathan Adams our Chief Scientist says:

“Without a meaningful way to order and categorise the information provided to the REF, the scholarly impact of many of these case studies may well have been hidden forever.  We have managed to produce order out of the chaos from thousands of unedited impact studies from different authors and hundreds of different institutions.”

The results show an unprecedented picture of the excellent outputs from colleges and universities throughout the country. The UK research impact is global and the REF analysis identifies a network of beneficial relationships with worldwide reach.

Jonathan Grant, Director of the Policy Institute at King’s College London adds:

“The overwhelming impression from our analysis of the impact case studies is of the quality, range and expertise of academic research in the UK and the benefits to communities around the world. Our analysis has yielded many fascinating observations and to our knowledge, is the first time that text mining has been used on such a scale to assess research impact. Now, with all the non-redacted case studies being made available, the potential for others to pick up and develop this work is significantly enlarged.”

[1] Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, Department of Education and Learning Northern Ireland.

The post UK universities launch REF Impact Case Study database to showcase global research outcomes appeared first on Digital Science.

REFlections is an an event for leading thinkers and policy makers in the world of research policy. The event is run by HEFCE and is taking place at the Royal Society in London on 25th March 2015. The event will provide an opportunity for attendees to hear about a package of evidence on the Research Excellence Framework 2014 including:

• Outcomes of the evaluation of impact in the REF from the perspective of submitting institutions and assessment panels
• Feedback from members of the assessment panels
• Emerging outcomes from the review of the costs, benefits and burden for institutions of REF 2014
• A first chance to access and use the online database of REF impact case studies, which has been produced as part of project undertaken by Digital Science in conjunction with The Policy Institute at King’s College London and Nature Publishing Group (see below for more info)
• Initial analysis of the impact case studies and what they tell us about the state of UK research
• Reflections on wider analysis and future work, including initial recommendations from the independent review of the role of metrics in research assessment

As part of the REF exercise, universities submitted case studies showcasing the impact of their research beyond academia. These studies outline changes and benefits to the economy, society, culture, public policy and services, health, the environment and quality of life. The goals of the analysis project undertaken by Digital Science, The Policy Institute at King’s and Nature Publishing Group are to:

• Make these impact case studies freely available in a form and format that will enable any researcher to carry out analysis, using a range of techniques and methods including automated text-mining. This will be achieved through an online database of case studies.
• Carry out a qualitative and quantitative analysis of the impact case studies, to extract common themes and messages that will form evidence of the broad impact of higher education research on wider society.

To find our more about the project to analyse the REF impact case studies read our blog post here.

Attendance is currently by invitation only but subject to availability of places, registration to the conference will be opened to others.  To register your interest in attending email admin@ref.ac.uk

The post REFlections: Evaluation of the REF 2014 and a Look to the Future appeared first on Digital Science.