roberts

http://www.hefce.ac.uk/pubs/hefce/2003/03_22.htm

Annex A

Review of research assessment: response form

Please complete and return as a Word attachment to an e-mail, to responses@ra-review.ac.uk. The deadline for responses is 30 September 2003.

Response by (name of person or organisation):

Standing Conference of Arts and Social Sciences

Corporate response (representing the views of the group or organisation): Yes
Private response (representing the views of one or more individuals): No

Contact in case of queries:

Name: Dr Sara Delamont AcSS
Tel: 029 20 874035
e-mail: Delamont@cardiff.ac.uk

Recommendation 1 (see paragraphs 113-116 of the review)

Any system of research assessment designed to identify the best research must be based upon the judgement of experts who may, if they choose, employ performance indicators to inform their judgement.

Do you agree or disagree with recommendation 1? Place a cross beside the appropriate answer:

Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree

x
...
...
...
...

Comments on recommendation 1:

In all comments the SCASS response draws on the One Voice document, a meta-analysis of the views of 27 organisations, and the views of its members.

This recommendation is strongly supported.

There is little or no support from within the Arts and Social Science community for the use of metrics, except as background information. In particular, there is no evidence that any such metrics correlate with research quality.

The recommendation to rely on expert judgement is the only way to undertake this task. There has been broad confidence in the expert judgements make in previous RAEs.

The idea of the peer is not simple: Panel members must have a range of relevant competences in pure and applied research, and have the trust of the disciplines.

Recommendation 2 (see paragraphs 117-126 of the review)

a. There should be a six-year cycle.

b. There should be a light-touch 'mid-point monitoring'. This would be designed only to highlight significant changes in the volume of activity in each unit.

c. The next assessment process should take place in 2007-8.

Do you agree or disagree with recommendation 2? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b		X
Point c				X

Comments on recommendation 2:

a)
The small increase in the interval between assessments is to be welcomed. This should have obvious implications for a pro rata extension to the 'census period' (i.e., the period for which research outputs may be submitted). This increase will make it possible to remove the difference in census period that currently exists between humanities and other subjects and will allow all subject areas to report on activity over a longer period. The gestation period of much book-based research (common in the humanities and in many areas of social science) is such that a longer census period will be very welcome and will allow more accurate assessment of research quality.

There would, in fact, be strong support for a cycle longer than six years, but we anticipate that government expectations make this unrealistic.

b)
It may be appropriate to monitor volume levels of activity at the mid-point of a cycle if the length of the cycle is increased. This should not be a mid-term mini-RAE.

Although paragraph 64 refers exclusively to reductions in research activity, there are no obvious reasons why increases in activity should not be investigated further.

(c)
The reasons for suggesting the next assessment in 2007-8 are well-known and relate to governmental requirements rather than to any specific requirements of the university system. There is a strong preference for 2008-9. Making an assessment in 2007-8, however means that there will be, in all probability, only four clear years between the finalisation of these consultations and the end of the census period. Even after consultations are complete, many procedural details will still have to be decided by the individual panels (see comments below). This means that the academic community will not know the rules under which it is to operate until a year or two before the assessment takes place. This is unacceptable.

The Chairman of the English Funding Council promised a five year gap of 'clear water', and the academic community can prepare for assessment and have confidence in the outcome only if there is such a gap between determining the rules and applying them. The obvious implication to draw is that a reliable and valid assessment could take place only in 2009-10. We doubt that this would be acceptable to the Government or the Funding councils, but we urge that proper consideration be given to the dangers inherent in an earlier assessment.

Recommendation 3 (see paragraphs 127-133 of the review)

a. There should be an institution-level assessment of research competences, undertaken approximately two years before the main assessment.

b. The competences to be assessed should be institutional research strategy, development of researchers, equal opportunities, and dissemination beyond the peer group.

c. An institution failing its assessment against any one of the competencies would be allowed to enter the next research assessment but would not receive funding on the basis of its performance in that assessment until it had demonstrated a satisfactory performance.

Do you agree or disagree with recommendation 3? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a	X
Point b	X
Point c			X

Comments on recommendation 3:

SCASS is not clear how far the procedures envisaged here would require HEIs to have strategies in Humanities and Social Sciences: many HEIs at present have policies for SET only.

SCASS is concerned that excellent research in Humanities and Social Science exists in HEIs with weak SET research, which should not be excluded from the RAE.

Recommendation 4 (see paragraphs 134-155 of the review)

a. There should, in principle, be a multi-track assessment enabling the intensiveness of the assessment activity (and potentially the degree of risk) to be proportionate to the likely benefit.

b. The least research intensive institutions should be considered separately from the remainder of the HE sector.

c. The form of the assessment of the least research intensive institutions would be a matter for the relevant funding council.

d. The less competitive work in the remainder of institutions should be assessed by proxy measures against a threshold standard.

e. The most competitive work should be assessed using an expert review assessment similar to the old Research Assessment Exercise.

Do you agree or disagree with recommendation 4? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b		X
Point c					X
Point d			X
Point e		X

Comments on recommendation 4:

(a) The three track procedure is to be welcomed, as it leaves the choice over submission to individual HEIs and their constituent Departments. It should not, however, be tied to absolute decisions on research funding that deny support for scholarship in predominantly teaching institutions.

(b) There is an assumption here that research quality and multi-tracking must be institution-level activities. There are strong research Departments in otherwise 'weak' institutions, and vice versa. The proposal would deny research funding and opportunities even to exceptionally strong Departments in institutions that are not 'research intensive'. If an assessment is to have any credibility it must operate principally at the Department/subject level, and not at the institutional level.

(c) This could produce 4 separate regimes in the 4 nations of the UK - which would be deplorable. The exercise must be UK-wide.

Recommendation 5 (see paragraphs 156-171 of the review)

a. The output of the Research Quality Assessment should be a 'quality profile' indicating the quantum of 'one star', 'two star' and 'three star' research in each submission. It will not be the role of the assessment to reduce this profile to summary metrics or grades.

b. As a matter of principle, star ratings would not be given to named individuals, nor would the profile be published if the submission were sufficiently small that individual performance could be inferred from it.

c. Panels would be given guidelines on expected proportions of three star, two star and one star ratings. These proportions should normally be the same for each unit of assessment. If a panel awarded grades which were more or less generous than anticipated in the guidelines, these grades would have to be confirmed through moderation.(1)

Do you agree or disagree with recommendation 5? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a					X
Point b	X
Point c			X

Comments on recommendation 5:

(c) SCASS members are divided on this. Some disciplines are strongly in favour, others feel it is a ridiculous idea. (For example, Social Policy favours it, Sociology is opposed).

SCASS's position is that the specialist sub-panels should award the grades they think correct, and if their percentages are different from those of their disciplinary 'neighbours' (as French was from Italian in 2001 for example), they should be asked to explain their rationale for the deviation.SCASS is cynical enough to think the media will always create league tables.

Recommendation 6 (see paragraphs 172-197 of the review)

a. There should be between 20 and 25 units of assessment panels supported by around 60 sub-panels. Panels and sub-panels should be supported by colleges of assessors with experience of working in designated multidisciplinary 'thematic' areas.

b. Each panel should have a chair and a moderator. The role of the moderator would be to ensure consistency of practice across the sub-panels within the unit of assessment.

c. Each panel should include a number of non-UK based researchers with experience of the UK research system.
d. The moderators of adjacent panels should meet in five or six 'super-panels' whose role would be to ensure consistency of practice between panels. These 'super-panels' should be chaired by senior moderators who would be individuals with extensive experience in research.

Do you agree or disagree with recommendation 6? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b		X
Point c		X
Point d		X

Comments on recommendation 6:

SCASS members are divided over the proposed two tier system. The central concern on which everyone is adamant is that academic research must be judged by experts in it: publications in Welsh cannot be judged by scholars of medieval Arabic, however distinguished.

(c): International experts need to be properly briefed on the RAE, and on the research in all 4 nations of the UK. They should see all submissions, and understand the consequences of the decisions for both future funding and for prestige.

Recommendation 7 (see paragraphs 198-204 of the review)

a. The rule that each researcher may only submit up to four items of research output should be abolished. Research Quality Assessment panels should have the freedom to define their own limits on the number and/or size of research outputs associated with each researcher or group.
b. Research Quality Assessment panels should ensure that their criteria statements enable them to guarantee that practice-based and applicable research are assessed according to criteria which reflect the characteristics of excellence in those types of research in those disciplines.

Do you agree or disagree with recommendation 7? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a					X
Point b	X

Comments on recommendation 7:

SCASS has no objection if SET disciplines wish to consider more publications, but in Humanities and Social Sciences four is appropriate.

Recommendation 8 (see paragraphs 205-213 of the review)

a. The funding councils should work alongside the subject communities and the research councils to develop discipline-specific performance indicators.

b. Performance against these indicators should be calculated a year prior to the exercise, and institutions advised of their performance relative to other institutions.

c. The weight placed upon these indicators as well as their nature should be allowed to vary between panels.

Do you agree or disagree with recommendation 8? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b		X
Point c				X

Comments on recommendation 8:

There is no simple way to work with subject communities in the Humanities and Social Sciences, especially in cross-disciplinary areas. There are at least 129 learned societies and subject defence groups. History has 4 bodies (at least). Time needs to be spent on the consultation which must be inclusively based.

Where a discipline values applied work it should be assessed properly.

Recommendation 9 (see paragraphs 214-234 of the review)

a. Where an institution submits to Research Quality Assessment in a sub-unit of assessment all staff in that sub-unit should become ineligible for the Research Capacity Assessment, even if they are not included in the Research Quality Assessment submission.

b. The funding councils should establish and promote a facility for work to be submitted as the output of a group rather than an individual where appropriate.

c. The funding councils should consider what measures could be taken to make joint submission more straightforward for institutions.

d. Where an institution submits a sub-unit of assessment for Research Quality Assessment, no fewer than 80% of the qualified staff contracted to undertake research within the sub-unit of assessment must be included in the submission.

e. All staff eligible to apply for grants from the research councils should be eligible for submission to Research Quality Assessment.

Do you agree or disagree with recommendation 9? Please indicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b	X
Point c	X
Point d			X
Point e	X

Comments on recommendation 9:

(a) The rules on this will need to be clearly specified well in advance of the census date for submissions.

(b) This is appropriate and unobjectionable.

(c) This is appropriate, but there should be no indication, express or implied, that joint schemes will be either advantaged or disadvantaged.

(d) SCASS members are divided on this point. In some HEIs this would work well, in others it could lead to the exclusion of excellent research from the scope of the panels.

Recommendation 10 (see paragraphs 235-238 of the review)

Each panel should consider a research strategy statement outlining the institution's plans for research at unit level.

Do you agree or disagree with recommendation 10? Place a cross beside the appropriate answer:

Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree

X
...
...
...
...

Comments on recommendation 10:

SCASS welcomes this. It would encourage HEIs to have strategies for their non-SET units, which not all HEIs currently do.

Question 11 Burden for institutions

The review proposals have been designed to make the burden of assessment proportionate with the possibility of financial reward. Do you agree that this has been achieved? Place a cross by the appropriate answer:

Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree

...
...
X
...
...

Comments on question 11 - burden for institutions:

It is not possible to tell until a new method is in place, and its impact has been evaluated by competent social scientists.

Question 12 Value of research assessment

What value do you place on the research assessment if the financial reward is likely to be small? Place a cross by the appropriate answer:

High
Medium
Low

...
X
...

Comments on question 12 - value of research assessment:

For the SCASS disciplines:

In Russell Group HEIs money and prestige are linked. In many post 1992 Universities and aspiring Universities, while the money is welcome, the scholarly validation of research is more important, especially if they are to recruit and retain good staff.

Question 13 Equality of opportunity for all groups of staff

The funding councils wish to promote equality of opportunity for all staff regardless of age, sexual orientation, political belief, disability, gender, race or religion and seek to ensure that its research assessment policies are compatible with this objective. How successful do you consider that the proposals of the research assessment review are in this respect? Place a cross by the appropriate answer:

Very successful
Successful
Neither successful nor unsuccessful
Unsuccessful
Very unsuccessful

...
...
X
...
...

Comments on question 13 - equality of opportunity for all groups of staff:

As the report itself states, there is little or no evidence on the impact of the RAEs since 1989 on equalities or inequalities: only assertion and occasional stories in the THES and the AUT Bulletin. We would regard the collection of data on these issues as a priority. Clear signals from panels that diversity of staffing will be rewarded are needed.

Question 14 Overall approach of the review

Notwithstanding your views on any specific recommendations, and given the responses to the earlier 'Invitation to contribute', do you agree or disagree with the broad approach taken by the review to the question of research assessment? Place a cross by the appropriate answer:

Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree

...
...
X
...
...

Comments on question 14 - overall approach of the review:

SCASS was disappointed that the review did not analyse the evidence submitted in 2002 by broad disciplinary areas, given the high degree of consensus in these responses. (The consensus is reported in the One Voice document available from SCASS).

A model that fits SET is not acceptable to the Humanities and Social Sciences, and that difference needs to be recognised and explicitly addressed.

Question 15 Further comments

Question 15 - any further comments:

The report shows little evidence of any use being made of the research base: the data gathered by social scientists on the RAE: policy change should be based upon data.

Note

1. This consultation question reflects an edited version of recommendation 5. The recommendation in the review report also states that 'the funding councils should provide institutions with details of the relative value, in funding terms, of one star, two star, and three star research, and of research fundable through the Research Capacity Assessment in advance of the assessment. These ratios might vary between disciplines.' In the event that the review recommendations are accepted, each funding council will develop its own policies for reflecting the assessment results in funding, taking proper account of Sir Gareth's recommendation.

Top of Page