Questions raised over State examinations marking process

Every year more than 100,000 students receive their Junior and Leaving Cert exam results.

The fact the exams are marked anonymously and major errors are relatively rare means they have widespread public confidence.

The State Examination Commission (SEC) says it is dedicated to providing a "high quality examinations and assessment system incorporating the highest standards of openness, fairness and accountability".

However, an internal report prepared by the commission raises urgent questions over whether the system really is fully transparent, fair and fit-for-purpose.

Twenty women-only professorships to be established this year

---

‘First step’ towards free books for primary school children

---

Hundreds of childcare providers may shut for ‘day of protest’

---

Government backs new €1bn Silicon Docks campus for Trinity College

---

Concerns

The findings are contained in an unpublished research paper, Standard setting procedures in the Leaving Certificate Examination.

The concerns stem from the measures used by the commission to ensure consistency of standards in students’ overall grades from one year to the next.

Nowadays, students are entitled to see their marked exam papers and appeal their results if they feel they have grounds for an upgrade.

This has been a big step forward in promoting greater transparency over candidates’ grades.

However, students do not have sight of the measures used to ensure consistency of standards in candidates’ overall grades from one year to the next.

The commission achieves this by adjusting marking schemes during the marking process to ensure a similar proportion of candidates are awarded As, Bs and Cs (or H1s, H2 and H3, as they are now known) each year.

As a result, it adjusts the marking schemes for individual exams.

This can involve diverting marks away from certain questions that are “easy” (if too many students are scoring high grades) or allocating extra marks for more challenging questions (if too many students are scoring lower grades).

When marks are diverted away from challenging questions, or towards easy questions, it has the effect of changing the weightings for different skills being assessed

The precise details of how this takes place has been hidden from public view. Requests by The Irish Times to see the details of how marking schemes are altered have been refused by the commission.

However, an internal research report produced by the SEC outlines several key ways the standard-setting process makes the marking scheme for exams “less valid” and reduces fairness by benefiting some students and penalising others.

1. Distorts the marking scheme
The report says a "critical disadvantage" of this approach is the "distorting effect" the procedure can have on marking schemes.

For example, when marks are diverted away from challenging questions, or towards easy questions, it has the effect of changing the weightings for different skills being assessed.

This discrepancy between the original marking scheme and the final one renders the exam “less valid that it would otherwise be”, according to the report.

2. 'Questionable fairness'
Changes to marking schemes vary from subject to subject – but in most cases it will change the "rank ordering" of candidates.

For example, let’s say John scores 92 per cent or a H1 under the original marking scheme, while Mary scores 88 per cent or a H2.

When the marking scheme is altered to get a similar proportion of grades to last year, John loses marks and ends up with 89 per cent or a H2, while Mary gains marks and secures a H1.

The report points out that changes to marking schemes affect this “ranking ordering”, which in many cases affects the grades students obtain.

These can be crucial differences for students who are chasing valuable CAO points for their chosen courses.

Significantly, a student will not know this has happened – even if they view their exam script; this is because only the final marking scheme counts and earlier versions are not published.

Much time and money goes on training hundreds of examiners to mark accurately using the original marking schemes

The report also notes the original marking scheme was designed to reflect, as accurately as possible, how skills being measured ought to be valued relative to each other. Therefore, any changes move it further away from this intended balance.

These factors and their impact on the rank ordering of candidates, the report says, is “more likely to reduce fairness than increase it”.

3. Distorts the management of the marking process
Ideally, the main focus of exam authorities is to ensure examiners are marking accurately and identifying any deficiencies.

However, the focus on the distribution of grades and the need to change marking schemes dominates the attention of authorities, according to the report.

The more time and attention devoted to this, the less time and attention that is available to focus on what should be their primary concern.

4. Compromises the accuracy of marking
Much time and money goes on training hundreds of examiners to mark accurately using the original marking schemes.

If these are altered, examiners are required to implement these changes without further training other than a phone call from their advising examiners. “This,” the report says,” hinders the accuracy with which the examiners can do their work.”

5. Slows down marking
The need to gather data from sample exam papers and determine whether changes are needed to the marking scheme takes a considerable amount of time towards the start of the marking process.

The report estimates it takes about five days to do this. Only after this, can the regular marking begin.

6. Impacts on the supply of examiners
Each year the commission struggles to hire enough examiners to mark hundreds of thousands of examination scripts.

The process of altering marking schemes – which involves suspending marking until sample scripts are analysed and revisiting papers which have previously been marked – is a source of frustration for many examiners.

Feedback from examiners who have declined to mark exams shows this is a factor in their decision not to take up an offer of reappointment.

7. Lacks precision and efficiency
The report notes altering marking schemes lacks "precision and accuracy".

For example, a marking scheme may be adjusted because it is proving too harsh by producing too few A or B grades, and too lenient at the other end of the spectrum by producing too few E and F grades.

The report notes adjusting marking schemes to address both of these difficulties at the same time is challenging.

8. Cannot facilitate 'linked' grades
Until recently there was no official link between grades at different levels in the same subject. This has changed since 2016 with the introduction of a new common points scale. For example, nowadays a H5 (50-60 per cent at Higher level) is the equivalent of an O1 (90 to 100 per cent at Ordinary level), which is reflected in CAO points. This link is now official and required under Department of Education policy. However, the report says, "under the present arrangement, "this is simply not feasible at all."

9. Limits adoption of new technology to improve marking quality
The process of adjusting marking schemes for standardising purposes is highly unusual internationally. This makes it more difficult to find compatible technology which allows for digitisation of scripts and on-screen marking.

Advantages

The report also examines the advantages of the existing standard-setting system used by the commission. Among these are that it is “familiar”; it “has the appearance of objectivity”; and may “contribute to keeping the standard of the paper consistent”.

The commission’s paper proposes two possible alternatives to the current standard-setting process, similar to systems used in the UK and elsewhere.

One involves setting grade boundaries after exams have been marked.

This would allow exam authorities to raise or lower the minimum marks needed to secure a grade, if an exam is easier or harder than the previous year.

This approach would ensure grades remain consistent year after year in a way in a way that is fairer for all candidates, as all candidates are affected in the same way.

Another approach involves applying a “standardising transformation”. This is a relatively complex process which typically involves converting marks into a uniform scale which gives equal value to work at the same level of performance, regardless of the level of difficulty of the exam. In simple terms, it transforms the marks of all students instead of moving the grade boundaries.

The research paper says the commission considers this approach would be “significantly better” than the current system.

“We recommend that the DES [Department of Education] approve the adoption [of] this procedure, following which the additional details can be discussed and agreed and an implementation plan developed,” it states.