The purpose of this document is to explain Umalusi`s quality assurance processes with a particular focus on the moderation of mark distributions in national examinations. This process is called `standardisation`. This document provides the rationale for standardisation and explains how it is done. Sharing this information forms part of Umalusi`s commitment to making its processes transparent to all who have an interest in the examinations Umalusi quality assures and certificates.
One of the responsibilities of Umalusi, as South Africa`s Quality Council for General and Further Education and Training, is to ensure that assessments and examinations of the qualifications it is responsible for are of the appropriate standard and quality. The qualifications currently quality assured by Umalusi are the following
- The National Senior Certificate (NSC
- The National Certificate Vocational NC(V)
- The National Technical Certificate - N3
- The General Education and Training Certificate (GETC)
- The Senior Certificate (SC) (the last SC examination will be offered in 2014)
These qualifications have prescribed components of both a) external assessment, which is nationally set, and b) internal assessment, which is set and assessed at the site level i.e. national examinations set by an assessment body and site-based assessments / continuous assessments set at the schools, colleges and adult learning centres.
The provision of national examinations is the responsibility of an assessment body, which may be an accredited private assessment body or the Department of Education. Assessment bodies implement various mechanisms, including the internal moderation of the standard of question papers, the training of markers and invigilators, the administration of examinations, checking of the marked scripts and internal moderation of the standard of marking, to ensure that the candidates` performances are fairly assessed according to the appropriate standard.
Umalusi, as the certifying body, quality assures examinations through the following processes:
- External moderation of the standard of question papers, to ensure that these satisfy the requirements as laid out in the curriculum and assessment policy documents of the qualification
- Monitoring of the entire examination process, including monitoring the state of readiness of assessment bodies to administer the examinations, the conduct of examinations and the marking of scripts
- External moderation of samples of marked scripts to verify that marking is consistent with the marking memoranda, and that marking is consistent across the various marking centres
- Moderation of examination and internal assessment marks to ensure consistency of standards
- Monitoring the resulting processes of assessment bodies.
Standardisation is the moderation process used to mitigate the effect of factors other than learners` knowledge and aptitude on their performance. The standardisation of examination results is necessary to take care of any variation in the standard of the question papers, which may occur despite careful moderation processes, as well as variations in the standard of marking that may occur from year to year. Other sources of variability include undetected errors and learners` interpretation of questions.
Standardisation is an international practice, and all large-scale assessment systems use some form of standardisation. The method used by the Cambridge International Examinations involves comparing the mean and standard deviations of the current exams with those of previous years. This data is then used to `set` the grade boundaries i.e. an A could be 80% and above in one year and could be set at 75% the following year, depending on the data. This system is also used by a number of African countries whose educational systems are still closely aligned with the Cambridge system.
The method used in South Africa is that of `norm-referencing`. The following sections outline the methodology and the principles underlying the methodology
One of the main assumptions underlying standardisation is that for sufficiently large populations (cohorts), the distribution of aptitude and intelligence does not change appreciably from year to year, i.e. one can expect the same performance levels from cohorts of roughly the same size across time. The standardisation process is based on the principle that when the standards of examinations (from one year to the next, or from one subject to another) are equivalent, there are certain statistical mark distributions, which should correspond (or be the same, apart from chance statistical deviations). Statistical moderation consists of comparisons between the mark distributions of the current examination and the corresponding average distributions of a number of past years, to determine the extent to which they correspond. If there is good correspondence, it can be accepted that the examinations were of an equivalent standard. If there are significant differences, the reasons for those differences should be established. On occasion, these differences may be due to factors such as a marked change in the composition of the group of candidates offering a particular subject, poor preparations for the examination because of some disruption in the school programme, or very good preparation because of special support from educators. In the absence of valid reasons for the differences, it should be accepted that the differences are due to deviations in the standards of the examination or of the marking, and the marks should be adjusted to compensate for these deviations.
Some examples of where mark distributions should correspond:
- For each subject with a large enough number of candidates, a desired or expected distribution of marks is determined, which is called `the historical average`. This is normally based on the raw marks (i.e. the unadjusted examination marks) for the examinations of the preceding three to five years. The distribution of the raw marks for the current examination is compared with the historical average
- The internal assessment (continuous or site-based assessment) marks of candidates should correspond with the adjusted examination marks obtained in the examination by the same candidates in the same centre. For example, if the average internal assessment mark for English in school A is more than 10% of the average adjusted examination mark for English in school A, then there is no correspondence and the internal assessment mark will be adjusted.
- The average of the marks obtained by the candidates offering a particular subject should correspond to the average of the marks obtained by the same candidates in other comparable subjects. It is expected that these two averages should be more or less be the same if the examination is of the appropriate standard. For example, for a common cohort of candidates, the average marks obtained in Mathematics should be close to the average marks obtained in Physical Science and /or other related subjects. If the difference in the averages is significant, it is an indication that the standards of the question papers probably differ.
Standardisation of results is achieved at a meeting between the assessment body and Umalusi. The assessment body presents its results after completing an analyses of its examination results, with a view to identifying any unexpected results, idiosyncrasies and cases deserving special attention. The results are also examined in light of interventions that have been implemented in the teaching and learning process, shifts in learner profiles, and so on. The assessment body makes sure that it has a thorough understanding of which adjustments will be appropriate, and what they would like to propose in this regard at the standardisation meeting with Umalusi. The standardisation process compares the statistical distribution of the raw examination marks of the current examination with the predetermined historical average distribution of the last 3-5 years, and considers the adjustments required to bring the distribution of raw marks in line with the expected distribution, taking into consideration the comparative subject analysis and moderation, and marking reports.
Umalusi will only consider adjustments where there is compelling evidence that it is necessary to do so, in which case the following may occur:
- If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards, subject to the limitation that no adjustment should exceed half of the actual raw mark, i.e. half of what the candidate got, or 10% of the maximum marks for the subject
- If the distribution of the raw marks is above the historical average, the marks could be adjusted downwards, subject to the limitation in a), above
Although it is clear that the statistical adjustment of marks to an agreed standard is by no means a perfect solution to the problem of year-to-year variations in the standard of examinations, it is currently more appropriate than a simple acceptance of the raw marks. Taken in conjunction with the rectification of cases where the deviations from the standard are excessive, it offers at least a partial guarantee of comparability between successive examination standards, thus giving candidates equal opportunity over the years, regardless of the standard of the question paper that the candidates wrote.
It must also be noted that currently examination test items are not pretested and calibrated. It is hoped that as the assessment systems start to use pretested items, the need for standardisation at the back-end of the examinations will be minimal. Finally, it must be stressed that mark adjustment cannot compensate for the effects of poor teaching or learning; its only purpose is to ensure equivalent standards are maintained across years for the different assessment bodies.