Final thoughts on REF2014 and UK Dental Schools

There has been relatively little critical analysis of the data from REF2014 compared to previous exercises.  While the reasons for this lack of interest remain unclear, it may be in part because individual Units of Assessment (UoAs) were so large as to make subject or institutional comparisons relatively meaningless.  The situation was made more complex by the different strategies adopted by institutions (and indeed different strategies developed by individual teams within institutions).  There have been very few criticisms of the large and often unusual combinations of subjects in different UoAs, either from the individual institutions, the wider academic community, or the media. It could therefore be assumed that the situation is generally approved or at least tolerated.  Inevitably though different research groups seek to compare their performance against peers, especially where critical analysis provides insight into best practice in terms of quality or impact.  The aim of this brief review is to determine how different institutions with Dental Schools performed in REF2014, and consider whether a more detailed and useful analysis could be performed on the basis of the data available.

The UK has 18 institutions with a Dental School or similar academic unit delivering dental education (see Table 1). Not all of these institutions made a return to UoA3, and even where they did it was not always possible to be certain that the institutional return included any staff active in dental research.  What is clear though is that the University of Liverpool, the University of Edinburgh, and the University of Aberdeen did not appear in the list of institutions that returned to UoA3, so in the event that dental and related research was submitted it would have been to a different UoA.  The remaining 15 institutions were listed in the UoA3 documentation available on the HEFCE website (http://www.ref.ac.uk/). 

Table 1:  UK academic institutions with a Dental School or related departmental activity alongside data from REF2014 for UoA3.  Note that this is the total number of staff submitted to UoA3 for each institution, the specific number of Dental School staff submitted to REF2014 is not known.

The summary data for UoA3 shows that 83 institutions made 94 submissions in total (as a number of institutions made multiple or joint submissions).  While this situation makes comparison more challenging, of the universities listed in Table 1 only four made multiple returns to UoA3 (Queens Belfast, Kings College London, Sheffield, and University College London).   The results of REF2014 were analysed by a number of independent parties including the Times Higher Education Supplement (THES), Research Fortnight, The Guardian, and a number of professional blogging sites such as Wonkhe (who utilised additional data made available by Exeter Consulting Ltd.).  Further data on staffing levels was made available after the release of the REF results by HESA, and this shed more light on differences between institutions. While few sources agreed on their detailed analyses, there appears to a be general consensus that (1) the data generated was probably less meaningful than REF2008, (2) the main winners of the exercise were the “golden triangle” institutions (especially large universities in London), and (3) of course the majority of individual institutions were able to generate “good news” headlines for their websites and press releases.  

Research Professional usefully combined the separate submissions made by individual institutions to UoA3 to produce a simplified institutional ranking i.e. each institution only appeared once in its league tables. This data was available on their secure website for subscribers at the time of writing, see (https://www.researchprofessional.com/media/pdf/UoA03_Allied_Health.pdf).  They weighted the data to create a number of useful metrics including Quality Index (a weighting based on the funding allocation, where 4* research is awarded 75% and 3* 25%), Power (how the Quality Index compares to the top performing University in terms of numbers of staff submitted) and Market Share (an estimate of the proportion of actual funding the institution will receive based on REF performance).  This data can be applied to produce league tables for institutions returned to UoA3 where it might be reasonably assumed that dental school staff contributed to the data.  Ranked by number of staff submitted to UoA3, Sheffield was one of the major contributors.  Again it is impossible to know how many dental staff were part of the overall figures (and even where a UoA3 return was named “dentistry”, this was misleading as it typically included staff from other health professions such as nursing or communication science).  By combining knowledge of the affiliation of UK Dental School with this combined REF data, it is possible to produce the institutional rankings given in Figure 1 below.

It may be argued that REF2014 was relatively unhelpful if one wanted to determine differences between specific subjects or departments as well as institutions (and therefore this was most likely the intention of those who were most closely involved in planning and execution).  Extending this argument, the only real beneficiaries were the “golden triangle” including the large London institutions, and it might also be concluded that they in some way disproportionately influenced the planning with this objective in mind.  It is therefore surprising how little public criticism there has been of REF2014 to date, with the few critical comments appearing to focus on the usual arguments i.e. how wasteful (in terms of time and resources) the whole exercise is, and how unhelpful the REF is as a tool to inspire and motivate academics to produce great research and scholarship.  That said, while my personal view is that there has been relatively little in depth or critical analysis of REF2014 compared to previous exercises, it should be acknowledged that a number of very good articles and blogs do stand out.  For example, the Times Higher Education Supplement has reflected some of the wider frustrations in the academic community, and John O’Leary produced a very insightful blog for Wonkhe (http://www.wonkhe.com/blogs/rankings-data-tables-and-spin/).  It would be wise for academics to promote ongoing discussion towards a fairer and more insightful analysis if this activity is going to be repeated in 2020, if the community remains silent we will undoubtedly see something that closely resembles the unremarkable REF2014.

With respect to the relative position of UK Dental Schools, at first sight it may appear impossible to compare performances using the results of REF2014.   Relatively few of the institutions returned in UoA had a dental school, some institutions chose not to return to this UoA, and others split their return across two or three submissions. Even where a split return was termed “dentistry”, an unknown proportion of the actual staff and subjects concerned would have been drawn from outside of the dental schools.  That said, almost all institutions and indeed dental schools have found some pieces of processed data to include on their web sites to demonstrate success in REF2014, but it is widely acknowledged this is related far more to marketing than any meaningful measure of difference.  Rankings of UoA3 based on power rating, then corrected for proportion of eligible staff returned, would provide useful comparisons but currently this is not possible (and it is unlikely to become possible in the future).  If it is essential that Dental Schools are compared, then this can only realistically be performed at the level of combined entries to UoA3 under power rating.  Sheffield lies 3^rd among institutions with a Dental School on the basis of both quality and power, in many ways a very satisfactory result. Ultimately though it is the financial benefits of a strong REF performance that matters most, and any artificial ranking will be less relevant in the absence of a cash reward.  The information that has emerged thus far suggests that this REF and subsequent adjustment of the national funding formula has been an exercise to further concentrate investment in London, Oxford and Cambridge at the expense of the N8, and northern Dental Schools could therefore suffer the ill effects of a badly executed REF for several years to come.  There now needs to be some response from the academic community who understand the value of an exercise performed well, and this includes the majority of UK Dental Schools who wish to maintain their important, distinctive and clinically relevant research for the benefit of the UK, including our industry and ultimately our patients.

Additional notes and data:  Research Professional also ranked institutional submissions to UoA3 by their Quality Index (Figure 2).  Differences in Quality Index were not as great as those seen in Figure 1 (above), with the greater discrimination by Power in part due to the number of institutional staff returned to this unit of assessment.  The different institutional policies are apparent here, for example KCL made a large return and benefitted in the Power Rating ranking (Figure 1), whereas Cardiff made a smaller return with a greater focus on quality, therefore coming 2nd in the Quality Index but falling to 5th in Power (it has been argued that as Cardiff are separately funded to the other UK institutions they were more interested in external perceptions of quality rather than the final HEFCE financial calculation).