Have you ever wondered how the Athletic Training Board of Certification Exam is actually scored? The answer is actually quite complicated (or simple depending on your perspective). The BOC exam is scored based on the Angoff Method, a widely used standard-setting approach in the test development world.
What is Angoff?
Angoff is simply a method of determining a cut-off score for an exam by having an expert panel judge each item’s difficulty. The cut-off score (or mark) divides students into two groups based on their performance; those below the cut-off and those above the cut-off. Passes and failures may be determined by the cut-off.
Cut-off marks are calculated using the Angoff method based on how candidates performed against a defined standard (absolute) rather than how they performed against their peers (relative). Exam items are judged (test-centered), not exam candidates (examinee-centred), and it is widely used for standardizing high-stakes exams. There are several variations of the Angoff method, but this blog will give you an overview of the process that most organizations use.
How is Angoff calculated?
According to Angoff, a group of experts is asked “What percentage of borderline candidates would answer this question correctly?” The experts must agree on the definition of a “borderline” candidate before making a judgment. For the Angoff cut-off score to be reliable, all judges must have the same definition of a borderline candidate. Judges should be subject experts since they must understand what makes a candidate competent enough to pass.
The expert’s judgments for an item should be the same or within a defined range (around 10%). Each item is assigned a predicted difficulty based on the average of everyone’s judgments. A cut-off percentage is calculated by adding up each predicted difficulty (mean) and dividing it by the total number of items in the exam. A cut-off mark is determined by the percentage of the total marks for the exam.
What if the experts disagree?
The experts discuss how they reached their decisions if their judgments are not unanimous. After the new judgments are made, the Angoff is recalculated. The process may be repeated several times.
Angoff method cut-off marks can be calculated as shown in the following table:
The average percentage for this example is 56.7. When rounded to 57, the cutoff percentage is 57%. The borderline candidate would be expected to get 57/100 marks if the test were out of 100.
Backing up your Angoff
To ensure the validity and reliability of the cut score, a number of other factors can be considered in addition to the judges’ predictions. To reinforce this method, a psychometrician will usually take a sample of past marks and the candidate’s expected results. The Angoff method can also be supported by using another standard-setting method, such as borderline regression, in order to provide results based on real candidate data. It may also be necessary to re-evaluate the standard-setting method if final candidate results do not reflect what should be expected of students taking the exam.
Why is the BOC Using Angoff?
Anoff is an established method for setting standards. As a high-stakes exam such as the BOC exam, it’s most reliable when supported by another method of standard-setting.
Advantages of using Angoff
Standard-setting based on Angoff holds up in court – The most widely used, formal method of standard setting is Angoff. The use of Angoff in high-stakes examinations is justified by the many published works on the topic.
The level of difficulty of the content – Angoff emphasizes just the content of the exam and how candidates should be performing at a certain level.
Once all judges have been trained, this is a relatively straightforward process.
Recyclable – If a question is re-used in another exam (in the same year group), the Angoff ‘predicted difficulty’ can be reused so that subject experts have fewer items to assess.
Disadvantages of using Angoff
It requires criterion-referenced methods to back up its accuracy and reliability since it does not use real exam data to estimate a cut-off mark.
It can be a long and labor-intensive process since the judges must examine every test item. Judges may become fatigued, impatient, and rush through the items as a result.
It is crucial for judges to have confidence – they must be experts in their fields. Judges must be confident and consistent when defining a ‘borderline’ candidate rather than just assuming an ‘average’ candidate.
For accuracy and reliability, you need a large sample of judges as well as people of different ages, genders, ethnicities, and levels of seniority. With such specifications, collecting a large sample can be difficult.
In a nutshell, Angoff is a standard-setting method that requires subject experts to predict what percentage of borderline candidates will get each question right by making judgments about its difficulty. There’s no doubt it’s reliable and holds up to scrutiny when challenged.