Understanding statistical moderation

The purpose of this document is to explain Umalusi`s quality assurance processes with a particular focus on the moderation of mark distributions in national examinations. This process is called `standardisation`. This document provides the rationale for standardisation and explains how it is done. Sharing this information forms part of Umalusi`s commitment to making its processes transparent to all who have an interest in the examinations Umalusi quality assures and certificates.

  1. Introduction

One of the responsibilities of Umalusi, as South Africa`s Quality Council for General and Further Education and Training, is to ensure that assessments and examinations of the qualifications it is responsible for are of the appropriate standard and quality. The qualifications currently quality assured by Umalusi are the following

These qualifications have prescribed components of both a) external assessment, which is nationally set, and b) internal assessment, which is set and assessed at the site level i.e. national examinations set by an assessment body and site-based assessments / continuous assessments set at the schools, colleges and adult learning centres.

The provision of national examinations is the responsibility of an assessment body, which may be an accredited private assessment body or the Department of Education. Assessment bodies implement various mechanisms, including the internal moderation of the standard of question papers, the training of markers and invigilators, the administration of examinations, checking of the marked scripts and internal moderation of the standard of marking, to ensure that the candidates` performances are fairly assessed according to the appropriate standard.

Umalusi, as the certifying body, quality assures examinations through the following processes:

  1. What is standardisation and is it necessary?

Standardisation is the moderation process used to mitigate the effect of factors other than learners` knowledge and aptitude on their performance. The standardisation of examination results is necessary to take care of any variation in the standard of the question papers, which may occur despite careful moderation processes, as well as variations in the standard of marking that may occur from year to year. Other sources of variability include undetected errors and learners` interpretation of questions.

  1. Is standardisation unique to South Africa?

Standardisation is an international practice, and all large-scale assessment systems use some form of standardisation. The method used by the Cambridge International Examinations involves comparing the mean and standard deviations of the current exams with those of previous years. This data is then used to `set` the grade boundaries i.e. an A could be 80% and above in one year and could be set at 75% the following year, depending on the data. This system is also used by a number of African countries whose educational systems are still closely aligned with the Cambridge system.

The method used in South Africa is that of `norm-referencing`. The following sections outline the methodology and the principles underlying the methodology

  1. What are the principles and assumptions underlying standardisation?

One of the main assumptions underlying standardisation is that for sufficiently large populations (cohorts), the distribution of aptitude and intelligence does not change appreciably from year to year, i.e. one can expect the same performance levels from cohorts of roughly the same size across time. The standardisation process is based on the principle that when the standards of examinations (from one year to the next, or from one subject to another) are equivalent, there are certain statistical mark distributions, which should correspond (or be the same, apart from chance statistical deviations). Statistical moderation consists of comparisons between the mark distributions of the current examination and the corresponding average distributions of a number of past years, to determine the extent to which they correspond. If there is good correspondence, it can be accepted that the examinations were of an equivalent standard. If there are significant differences, the reasons for those differences should be established. On occasion, these differences may be due to factors such as a marked change in the composition of the group of candidates offering a particular subject, poor preparations for the examination because of some disruption in the school programme, or very good preparation because of special support from educators. In the absence of valid reasons for the differences, it should be accepted that the differences are due to deviations in the standards of the examination or of the marking, and the marks should be adjusted to compensate for these deviations.

Some examples of where mark distributions should correspond:

  1. How is standardisation achieved?

Standardisation of results is achieved at a meeting between the assessment body and Umalusi. The assessment body presents its results after completing an analyses of its examination results, with a view to identifying any unexpected results, idiosyncrasies and cases deserving special attention. The results are also examined in light of interventions that have been implemented in the teaching and learning process, shifts in learner profiles, and so on. The assessment body makes sure that it has a thorough understanding of which adjustments will be appropriate, and what they would like to propose in this regard at the standardisation meeting with Umalusi. The standardisation process compares the statistical distribution of the raw examination marks of the current examination with the predetermined historical average distribution of the last 3-5 years, and considers the adjustments required to bring the distribution of raw marks in line with the expected distribution, taking into consideration the comparative subject analysis and moderation, and marking reports.

Umalusi will only consider adjustments where there is compelling evidence that it is necessary to do so, in which case the following may occur:

  1. Conclusion

Although it is clear that the statistical adjustment of marks to an agreed standard is by no means a perfect solution to the problem of year-to-year variations in the standard of examinations, it is currently more appropriate than a simple acceptance of the raw marks. Taken in conjunction with the rectification of cases where the deviations from the standard are excessive, it offers at least a partial guarantee of comparability between successive examination standards, thus giving candidates equal opportunity over the years, regardless of the standard of the question paper that the candidates wrote.

It must also be noted that currently examination test items are not pretested and calibrated. It is hoped that as the assessment systems start to use pretested items, the need for standardisation at the back-end of the examinations will be minimal. Finally, it must be stressed that mark adjustment cannot compensate for the effects of poor teaching or learning; its only purpose is to ensure equivalent standards are maintained across years for the different assessment bodies.