The Standing Conference of Arts and Social Sciences (SCASS) was founded in 1984, and represents faculties of Arts and Social Sciences in HEIs. SCASS welcomes the opportunity to contribute to the review. The response is structured to address the issues raised in Annex B, with additional points made towards the end, as requested.
This response is from the Executive Committee of SCASS. We are happy for it to be published. Queries should be addressed to the Convenor:

Dr Sara Delamont, AcSS
Cardiff School of Social Sciences
Cardiff University
Glamorgan Building
King Edward VII Avenue
Cardiff CF10 3WT

Group 1: Expert Review

5. Combining assessment of teaching and research, while reinforcing the essential ties between them inside HEIs and in the public mind, would be problematic for at least five reasons.
(a) The different nations within the UK have had different systems of assessing teaching, and Scotland, particularly, would be unwilling to abandon its ‘‘own’ procedures. That could threaten the essential UK-wide nature of research assessment.
(b) Funding streams for research and teaching have now been carefully separated. As a result, different HEIs have such different missions that devising one system to cover a ‘teaching only’ HEI and a ‘research only’ organisation would be a Herculean task.
(c) The criteria for satisfaction or excellence in teaching and in research are so different that one system seems unworkable.
(d) At this stage, collapsing the two modes of assessment would risk producing a fudging figure that could enable Government to claim all was well everywhere in the university system, when in fact many institutions are struggling to keep up and develop their research with totally inadequate funding, and teaching overload.

(e). Separate assessments of research and teaching would still be required, so this exercise would not reduce the burden of paperwork.
6. SCASS is acutely conscious of the problems that can arise with expert or peer review, but are is absolutely convinced that other systems would be worse. Expert review should be retained: but the burden on the chosen ‘experts’ needs to be reduced, and the use of international experts needs further discussion.
7. (a) A combination of prospective and retrospective assessment is desirable, but a rebalancing so that more emphasis is placed on the prospective.
(b) Data provided in last RAE were fine. More information on the funding context of the research should be provided, to give assessors a more informed insight into the conditions of research production and consequently a base from which to evaluate whether improved funding might bring developing excellence to fruition.
(c) Assessments should be of individuals working in their real immediate group context. Nomination of that context should be devolved to the nearest point to the workface within individual institutions. This will generally be departments, but a major growth in research groups or centres within institutions is an increasing possibility. (This is the effect of the split in research-teaching funding, and there is now no option but to live with it.)
If interdisciplinarity is to be encouraged, the submission of units wider than departments has to be possible; if regional or other collaborations are to be encouraged, then cross-institutional submissions have to be possible. The individual or the research group is not appropriate if the funding stream is to the institution. In the majority of cases current UOAs are about right.
(d) Subjects or thematic areas allow appropriate experts to be appointed: the current range of UOAs already forces some researchers into inappropriate areas. If any change in the number of UAS is contemplated, there should be more, not fewer. Some UOAs are currently too diverse, and need to be further divided and made more specialised. Radical changes to the base of assessment would be unhelpful, if not actively damaging. The ability to measure improvement in standards over time would be lost. The research community would be forced to waste more valuable research time trying to master a changed bureaucratic system and a new rhetoric.
(e) The major strength is transparency: it is easy to recognise what UOA rating relates to which department, and to determine who is, and is not an appropriate ‘expert’ in that discipline. There are problems in some disciplines with the concept of ‘international’ research: this is easier to define in, for example, physical chemistry than in the Celtic languages or Welsh History.

Group 2 Algorithm

8. It is entirely mistaken to believe than any algorithm is objective. The work of the sociologists of science since the 1960s has repeatedly demonstrated the underlying subjectivity behind such measures. SCASS can, if necessary, provide expert advice for the committee to show that none of the five suggested ‘objective’ measures is actually any such thing.
(a) Reputational surveys are the most unreliable and invalid metric of the five suggested, and SCASS is totally opposed to their use.
(b) Research Income is an input measure, and research assessment should be a judgement of output and performance.
(c) Bibliometrics are unreliable in all disciplines that proceed by book publication rather than journal publication: so in humanities and most social sciences they are distrusted, because monographs and edited collections contain much of the best research, missed by bibliometrics. Citations are particularly unreliable in disciplines where new ideas are presented in books.
If bibliometric judgements are to be made about books, a particular difficulty arises. Currently panels in humanities and social science look for monographs produced by high status publishers. The commercial decisions of a diminishing number of multinational publishers determine what monographs are published. Books issued by smaller houses, which have smaller promotional budgets, are less likely to be known, and therefore cited, so a bibliometric, citation based system would doubly disadvantage than those who must publish with smaller houses because their work is specialised or judged not to be commercial.

(d) Research student numbers, and competitions, should be one metric, but need to be interpreted. Several research councils, for example, the MRC allocate students on the basis of past research income, not current excellence. In humanities and social science, unlike sciences the presence of a group of research students is not necessary for productive staff to advance their ideas
(e) We are unsure what is meant by measures of financial sustainability. If it is proposed that HEIs are required to demonstrate that the QR monies earned in previous RAEs have gone into the UOAs that earned them, this would be welcomed by many social science departments, but would be unpopular with VCs.
9. We would advise the councils to examine the social science evidence on the lack of objectivity in all metrics.
10. (a) No - absolutely not
(b) None that are objective
(c) No
(d) The metrics would immediately be distorted by ingenious strategies. They are not reliable now, but their reliability would decline.
(e) There are no strengths. The weakness is the mistaken idea that metrics are objective when they are not.

Group 3: Self Assessment

12 and 13
(a) In a genuine self assessment, HEIs would have to have freedom to include any data they chose. However such a self-assessment would have to include lists of publications, grants, and strategies for developing and sustaining research cultures. If the system were changed to one in which the 2001 gradings held for longer periods (e.g. 10 years or 15 years) on production of a basic self assessment, SCASS would not object. However there would have to be a system of checks on a sample of self-assessments, and HEIs would have to be able to request a full review after five years. So a panel would be needed to judge the sampled self-asessments and the regrading requests.
(b) A combination of both, with a bias to the prospective.
(c) If self assessment were to be used, expert panels in each discipline would have to produce a list of minimum criteria to be included in self assessments in that discipline.
(d) There would have to be random probity audit of factual material and random evaluation of claims to maintain existing standards, as well as automatic detailed scrutiny of self-assessments that claimed improvement on previous performance. All departments with 5* ratings should automatically undergo full panel scrutiny. With these safeguards, the whole exercise would be much less expensive in terms of money and staff time, but should still command respect.
(e) Self assessment of a routine kind to maintain an existing rating would be less burdensome. Claims for a regrading would be burdensome on those institutions, but that would be a reasonable burden because voluntarily requested.
(f) Self assessment could not be the basis for substantial redistribution of QR monies: for maintenance of a grade given by an expert panel, they would be acceptable if a good audit system existed.

Group 4. Historical Ratings

16 (a) We are unconvinced that the distribution of research strength in most humanities and social sciences changes slowly. In science and engineering where patterns of grant income, equipment, and large research groups, do indeed change slowly, if at all, the distribution of research strength is seen by experts as relatively stable. In most humanities and social sciences, research strength can change quite rapidly. Sociology at Cardiff, for example got a 2 in 1989, a 3 in 1992 a 4 in 1996 and a 5 in 2001 and this is not an isolated example. If the USA recruits the best humanities or social science researchers from a department the distribution of strengths changes very quickly indeed.
(b) The 2001 RAE provides a reasonable baseline.
(c) Only regular review by expert panels can reliably identify rising and falling UOAs. A value for money element would need to include data on the internal distribution of QR monies, within the HEI.
(d) There is relatively little research on the effects of the RAE on the behaviour of academics and of HEIs. There are rumours, but not evidence. A greater reliance on historical ratings would probably stifle innovative work in humanities and social sciences, because in these disciplines advances came from all types of HEI and all regions of the UK.
(e) The major weaknesses of historical ratings is stagnation and complacency. In humanities and social sciences, where research is relatively cheap, it is very important to recognise what HEIs are doing to develop research ‘against the odds’. Credit should be given to UOAs who have invested in the research careers of younger staff.

Group 5: Cross Cutting Themes

18 a. Once the AHRB has become a funding council, greater collaboration between the funding councils and the research councils to minimise the burden would be welcome. The social sciences, apply every five years for postgraduate recognition from ESRC. The data required are burdensom to collect and submit, and they duplicate several aspects of the data required for RAE. If the research councils and the research councils could agree on collecting the same data in the same format over the same periods, burdens would be reduced. However the severe understaffing of the research councils would need to be remedied so adequate databases can be maintained.
18 b. An interval of ten years between routine RAEs (with scrutiny of 5* UOAs, and a voluntary request system every five years) would be sufficient. The time scale for counting publications should be seven years for both humanities and social sciences, rather than seven and five years as at present - time to settle into a production mode and process material through the publishing pipeline. A single big-bang assessment will minimise institutional disruption, and make it possible for cross-disciplinary submissions to be made easily.
18 c. The ‘best’ research is innovative in, for example, subject matter, methodology, applicability. ‘Good’ research might not be innovative, but could, for example, be doing necessary follow-up work in an area developed by someone else. The standards established in RAE 2001 seem to have gained general acceptance.
(d) Providing QR monies to a subject on the basis of its ratings is problematic, especially because of the risk that panels might inflate their ratings to increase the ‘subject pot’. On the other hand, the historical balance reflects costs not merit. A strategic judgement on the importance of the subject to the UK is too open to political pressures and biases. SCASS’s origins lie in the defence of the humanities and social scieces against precisely such ill-informed political prejudices in the early 1980s.
Criteria such as external funding and international competition can be problematic (as we have suggested above) some specialist areas. For example, research on Scottish politics or social history does not attract large funds, and its international status could be hard to assess, but its quality can be outstanding. Historical allocations are unfair to new and rapidly expanding subjects, or those where innovations demand new levels of funding: the use of CIT in humanities is an example of new techniques needing new money.
18 e. If it were possible to devise a ladder of opportunity it would probably be welcomed in the humanities and social sciences community.
18 f. We welcome discussion of the possibilities of grouping subjects - perhaps into cognate areas corresponding to research councils - and developing assessment in different ways for each area. However, this would reduce the transparency of the process and the results for users outwith the academic community.
18 g. If the QR monies are given to HEIs, then, realistically, they have to be ultimately responsible for their submissions. However it is arguable that UOAs rated 4 or above should be required to return 100 per cent of staff, to stop the practice of trading funding for the prestige of a high rating. That would prevent the damage to individuals, recognise different HEI missions, remove the superscript qualification to the higher grades, and prevent discrimination. (see below)
18 h. We are not convinced that sufficient research has been done to substantiate the accusations of discrimination by HEIs against some categories of staff. However, the rumours and accusations are themselves damaging to the aims of the funding councils. We see a place for explicit statements from the funding councils that expert panels in humanities and social sciences will not downgrade UOAs who include, for example, gay and lesbian staff researching gay and lesbian issues.
Alternatively, a requirement to return 100% of staff could help ensure that all those in post are returned even if they were engaged in controversial research areas such as Queer Theory. However such a requirement might militate against controversial researchers being appointed to HEIs in the first place, which would stifle intellectual innovation.
18 i. We regard the 3 most important features as:
rigour
transparency
resistance to political interference.

Group 6: Omissions

19. A. SCASS wishes to emphasise that the evaluating ‘experts’ must primarily be academic peers. The presence of too many ‘stakeholders’ with their own notion of what research is ‘for’ will slew the exercise in utilitarian directions. The capacity for Blue skies research and respect for the primacy of academic values must be maintained.
B. Fuller understanding is needed of the impact of research funding, or its absence on the regions and communities where institutions are sited.

C. Account should be taken of the relation of the standard of work produced to the levels of funding and availability of resources in institutions, which differ substantially (time, library access, access to networking opportunities, contacts, etc.). This should however be a separate exercise from the assessment of quality: it relates to the allocation of funding.

D. Factors A and B can only be taken into account with great care. Otherwise, there is a risk of building into the exercise a presumption in favour of established areas, and of departments and institutions who can afford to run training programmes out of their own funding resources. In Humanities and Social Sciences, new ideas often come from the edges, and it is important to remain alert to the need to sustain the potential for research throughout the system.