NEWS
Case study about the development of MarkManager's first version
Mitra Bhar, Manager IT at NESA, says "It is important that we continue to investigate how advances in technology can provide improvements in our examination and assessment programs. LANSA's technology and the IBM i platform help us to provide accurate, highly reliable and secure solutions to meet the requirements of our core business."
"LANSA's technology and the IBM i platform help us to provide accurate, highly reliable and secure solutions"
- The Logistics of Exams
- Marking Procedures
- Onscreen Marking
- The Benefits
- Conclusion
- Company and System Information
The Logistics of Exams
The two main qualifications which NESA administers are the School Certificate (SC) and Higher School Certificate (HSC). In recent years over 87,000 Year 10 students took part in the SC examinations and over 68,000 Year 12 students in the HSC examinations. Most of these students take at least six examinations. Recent HSC exams included over 110 different examinations for a wide variety of subject courses. Students worked from 13 million printed pages of questions and submitted their answers in 1.75 million writing booklets.
The setting, conduct and marking of the examinations is a huge logistical task, which involves over 5,000 examination supervisors at 750 exam centers, 7,000 markers assessing practical and written examinations at over 20 marking centers across the state, and more than 750 casual clerical staff.
The appointment system for exam supervisors and markers and the collection of the actual marks has been managed with a LANSA-based solution for many years, but the marking process itself, except for multiple choice exams, was mostly paper-based.
The marking process consumes a significant proportion of NESA’s annual budget. Accordingly, streamlining the marking process can have a major effect on the NESA’s cost-effectiveness.
"The setting, conduct and marking of the examinations is a huge logistical task"
Marking Procedures
In order to assure a fair outcome for all students, the marking process for written exams is highly structured and sophisticated. The process involves chief examiners, who have been involved in designing the exam, supervisors of Marking, who manage the marking operation in a subject marking center, senior markers, who are typically teachers with extensive marking experience and markers, who are experienced teachers or academics.
After being briefed about marking guidelines by the chief examiner and supervisor of marking, the senior markers will develop a marking kit for the exam section they are responsible for. A section can consist of one or more questions. There are usually at least two senior markers for each exam section. To develop the kit, they first read through a large number of actual scripts (the written responses by students) to get a feeling for the exam at a practical level.
The marking kit needs to include at least one complete set of scoring benchmark examples, such as scripts scoring ‘5 out of 5', ‘4 out of 5' and so on. The kit also contains detailed marking guidelines and sample markings. The kit is then tested by other senior markers, again on a representative sample of actual scripts. During this process ‘marking reliability statistics' are calculated to demonstrate whether a kit is working well, or whether further fine-tuning is needed. None of the marking during this process, which may take several days, is final.
Next the markers arrive at the marking centers to prepare for and do the real marking. Again, each marker is responsible for only a specific question or section of the exam. Markers first need to go through a number of ‘practice scripts' to get them on the same level of marking as the senior marker has set out. When a marker consistently marks in the expected range on the practice scripts, he or she is ready to start marking real scripts.
The senior marker, who looks after a team of 5 to 10 markers, will randomly insert so called ‘common scripts' in between the real scripts, which serve to check and make sure that the markers stay fresh and accurate and keep marking within the intended range.
Extended responses and essays are marked independently by two markers and the final mark is the average of the two. If the two marks vary by more than one third of the maximum mark, a third marking will be undertaken, and a senior marker will determine a 'resolved mark'.
"The marking process for written exams is highly structured and sophisticated"
Onscreen Marking
Muir Mathieson, Senior IT Consultant, NESA, and technical project leader for the Onscreen Marking System, explains, "The onscreen marking process begins with the color scanning of the writing booklets or question and answer booklets the students have submitted and, using LANSA's integration tool (LANSA Integrator), upload them in PDF format into the IBM i-based marking system."
"In that process, a custom written LANSA Integrator PDF service takes care that the number of pages is validated, that student and exam center identification details are masked and that the page images are reassembled as required for marking, so that a marker only gets to see the exam section he or she needs to mark. The LANSA Integrator service also sets the PDF properties and saves the documents to their proper file name."
Authorized markers can access the scanned scripts remotely over the Internet, or from one of the marking centers, with the NESA browser-based Onscreen Marking System, developed with LANSA high level language. The system has specific functionality for each of the main roles associated with the marking process: markers, senior markers and supervisors of markers.
The system allows markers to set preferences for their comfort based on the equipment they are using, such as left or right mark boxes or buttons. Markers can view a list of all the scripts they have been working on that day or previous days. They can add notes for themselves or for immediate attention of their senior marker. They also have the ability to search through all the scripts they have marked using several different search criteria, making it very easy to locate scripts they need to revisit. The system navigates the markers efficiently through the work that needs to be marked, automatically and randomly serving up the next script.
Senior markers have additional functions. They can annotate and upload benchmark examples and practice scripts, and set up a frequency for inserting common scripts in between the real scripts. They have a real-time dashboard view of what their markers are doing and can go to any script to look at the marker's notes and marks awarded.
"Marking deviations in the practice and common scripts are shown in color, making it easy to discuss variations and interpretation of the marking scheme with that marker. Markers who are consistently ‘on-target' need not be distracted or taken off the task. Similarly, it's also easy to supervise new markers and decide whether they need to be given more practice scripts or have them start on the real scripts," says Mathieson.
The system allows markers and senior markers to exchange messages. This is helpful when a marker needs assistance or wants the opinion of the senior marker about a certain script. Likewise, the senior marker may feel the need to comment on the marking of a certain script and send an instantaneous message to the marker. The messaging system is especially helpful for remote marking, allowing teachers to mark at home.
At any time, the senior markers can view statistics. They can see how the spread of markings for their exam section is evolving, how many scripts are marked and still need to be marked, how much time markers spend on average and on individual scripts.
All internet traffic is encrypted using SSL 128-bit encryption and no marks ever reside on the marker's desktop. All data (including scripts) are replicated across to a second IBM i server in real-time with each server residing in separate data centers.
Not directly related to the Onscreen Marking System, but part of the same exam system, is the digital recording of oral language exams. This is still at a trial stage for main languages, but already implemented for some language exams. At the point of examination the student sits in front of a computer that is connected to the NESA’s examination system. After photo identification, the student's response is digitally recorded and for some exams, marks are entered immediately. Once the exam is over, the examiner submits the digital recording and marks to NESA’s IBM i-based exam system, where LANSA Integrator takes care of the receiving, naming and storing of the file in MP3 format. The digitally recorded exams can then go through a similar marking process as the written scripts, using exactly the same application with modified terminology.
"Using LANSA Integrator, we have been able to create a highly sophisticated application, where PDF and MP3 files are fully integrated with our IBM i-based core examination system," says Mathieson.
"We have created a highly sophisticated application, where PDF and MP3 files are fully integrated"
The Benefits
Kevin Ford, Manager, Student Support Services Branch at NESA , says, "The Onscreen Marking System provides instantaneous marker reliability data and promotes objective and consistent marking. The same strict procedures apply to both paper-based marking and onscreen marking, but the procedures are easier to apply and manage in the onscreen solution."
"Having a real-time view of marking and having the tools and statistics available to spot deviations, allows senior markers to take immediate corrective action. In the paper-based system it could happen that a large batch of scripts would have to be re-marked, as it could take a while for the senior marker to notice that someone was marking outside the range," continues Ford.
"Another advantage is that the script and notes can be separated, which is good when a script needs to be objectively re-marked. In that case you don't want a marker to be influenced by the notes and comments made by someone else. Likewise, the fact that the marker only gets to see the section of the exam that he or she is responsible for, also encourages objective marking."
The common scripts are more effective in the onscreen marking process, because markers cannot differentiate them from the real scripts. In the paper-based system you could recognize them, because they are photo copies while the real scripts are originals. "Although it should theoretically not make a difference to the markers that they are marking a common script, in reality they may give that script extra attention, because they know it will be monitored. Now, every script is marked entirely on its own merits, making for a fairer and more equitable outcome for all students," says Ford.
Not being dependent on paper-based procedures allows NESA to draw on a larger pool of eligible markers. It is also very useful for teachers to gain practical experience with the marking process, so they can better prepare their students for the exams. As traditional paper-based marking requires markers to come to a marking center, the distance made it impractical for many country teachers to participate. "Now we can give them Internet access and support them with online practice scripts, marking kits and messaging. We can even involve teachers who teach the HSC at overseas schools and colleges," explains Ford.
"The markers are extremely positive about onscreen marking, because of the convenience, the potential for reducing the movement of large quantities of paper and the quick feedback on their marking," says Ford. "There is also a firm belief that the system is fairer and more equitable for the students."
"The logistics involved in securely distributing, storing and collecting nearly two million paper-based writing booklets are staggering," concludes Ford. "It is a far neater and easier-to-manage process to provide secure access to scripts that are kept on our own server. The solution is also more cost effective."
"The Onscreen Marking System provides instantaneous marker reliability data"
Conclusion
The Onscreen Marking System is part of a wider eAssessment initiative. The eAssessment also includes a Computers-in-Exams (CIE) initiative, which allows for exams to be conducted and marked online, eliminating the need for hand-written responses altogether. Some of the HSC computing exams are already conducted online. When given a choice, a significant proportion of students would prefer to use a computer for their exam rather than hand-writing.
NESA has conducted extensive trials for all eAssessment initiatives, comparing paper-based marking with onscreen marking and handwritten scripts with online exams. Reactions to the CIE initiatives have been very positive from students, teachers and markers.
Bhar, says, "We have tried our Onscreen Marking System with the exam papers from schools in other states and other countries and also with the examinations of universities. The system works equally well in those scenarios."
"Looking at the international experience, onscreen marking and e-Assessment are the way of the future, It is important that we continue to investigate how technology can provide improvements in the assessment and examination programs."
"The services we provide are highly visible and critical to the smooth administration of our examinations. We cannot afford to make any mistakes. LANSA's technology and the IBM i platform help us to provide accurate, highly reliable and secure solutions to meet the requirements of our core business. With a small in-house team and using LANSA's productive high level development tool, we can support a system that can be used by virtually all teachers and students in NSW or elsewhere," concludes Bhar.
"With a small in-house team we can support a system that can be used by virtually all teachers and students in NSW or elsewhere"
Company and System Information
- The NSW Standards Authority (NESA) is the certifying authority responsible for the content of syllabus materials, registration of non-government schools, and delivery of high school examinations for New South Wales, the most populous state in Australia. Using LANSA since 1989, the NESA development team has delivered many systems, such as online services to schools, teachers, students and parents. These include the delivery of HSC results over the Web and the collection of over 1.2 million grades and assessments and over 45 million responses and marks per year.
- NESA uses an IBM Power 570 with the IBM i operating system.
- With advice and assistance from LANSA, NESA has commercialized its Onscreen Marking System, now known as "MarkManager". Speak to your LANSA Account Manager, or email LANSA for more information.
- For more information on NESA visit: https://educationstandards.nsw.edu.au/wps/portal/nesa/about/who-we-are/our-story