How does it work?
MarkManager presents candidate responses on-line for marking.
It also automates the surrounding workflow to provide an end-to-end assessment solution for certifying authorities, awarding bodies, testing organisations and other educational institutions.
MarkManager’s architecture supports web-based access to reliable and highly scalable application servers. Only a supported web browser and Adobe Reader is required on the client device. This “zero deployment” model simplifies implementation and extends application reach.
The solution can be securely accessed as a hosted service, under the “Software as a Service” (SaaS) model, or licensed on-premise. MarkManager as a service supports both multi-tenanted and fully virtualized implementations. Script scanning can be included in one overall service to provide a complete solution.
Marking of each course within MarkManager is an independent exercise. For any given course the marking process has two main phases, the first preparatory and the second beginning as soon as the examination scripts are collected and assembled from exam centres.
MarkManager supports both paper-based scripts and responses entered on-line by candidates and audio recordings. In a controlled on-line environment, markers access the scripts remotely over the Internet for marking.
With MarkManager, the main stages for marking each course are as follows:
MarkManager provides comprehensive support for the auditable workflow management of the marking process throughout the entire marking cycle. The processes within the cycle are as follows.
1 Configure Course
A panel leader (“Supervisor of Marking”) determines a marking strategy for the course, and reflects that against a scanned image of the course’s master script. Allowable mark ranges are specified by question. Specialised or split marking (segmentation of the script into answers to be marked by different marking teams) is reflected against the script, which is also masked and cropped as required, e.g for anonymity of the candidate and the examination centre.
MarkManager empowers the customer to configure each course using internal resources.
2 Configure Marking Teams and Candidates
Marker and candidate data is uploaded via a standard interface with the customer’s Examination System. The panel leader assigns a profile to each senior marker and marker, determining for example which questions can be allocated to the marker and how many logon scripts will be presented.
3 Scan and Upload Scripts
MarkManager is compatible with industry-standard scanning equipment for preparation and upload of completed scripts, tagged by identifying data. Script page numbers are automatically checked to verify scanning integrity. Each script is allocated a unique random number as its identifier for control purposes and so that scripts can be presented in a random sequence at marking time. An option exists for scripts to be assembled from on-line answers entered by candidates, if applicable. For examinations with spoken responses digital recording upload is supported.
4 Pilot Marking
Senior markers develop marking benchmarks and guidelines for each course. If applicable under your policies, senior markers first carry out wide reading of completed scripts and then carry out test marking before categorizing selected scripts for subsequent reference by MarkManager.
Available script categories in MarkManager include:
- Wide reading
- Benchmark (example for marker reference of specific mark allocation)
- Practice (for marker familiarisation and training)
- Logon (start-up script after marker’s logon), and
- Quality assurance (seeded script to check marker accuracy).
5 Marker Training
Markers review the marking scheme and guidelines, before conducting wide reading, followed by practice marking. When a marker consistently marks in the expected range as set by the benchmarks, he/she is authorised to proceed to actual marking.
System-wide and course-level parameter settings determine the policy to follow at this point of the cycle.
6 Live Marking
After each logon by a marker, he/she must first correctly mark the designated number of logon scripts before proceeding. Markers can set display preferences to suit their comfort and working style. MarkManager ensures marks awarded for each question are within range, and that no responses are missed, via a “non-attempt” facility. Pre-marked quality assurance scripts are seeded into the workload to verify continuing marker performance to required levels of consistency. Markers can electronically annotate a script with comments and place scripts on hold for referral to a supervisor. Workflow support ensures supervisor and marker collaboration is optimised. Daily and total quotas apply and can be adjusted when necessary. A marker can search scripts he/she has already marked on certain criteria, for example by range of marks for specified question parts, if marks previously awarded need to be reviewed.
Double marking is carried out to the ratio specified in the marking scheme. Scripts for double marking are presented in random sequence to a second marker who does not have access to earlier marks and annotations. Such blind marking preserves the integrity of the marking process. MarkManager then automatically determines whether the difference between marks constitutes a discrepancy, and actions accordingly.
Supervisors monitor marking as it is conducted and have a real-time dashboard to report on marker performance against logon and quality assurance scripts and to highlight any unusual pattern in the range of marks a given marker has awarded. Supervisors can view any script of any marker and the marks awarded. Corrective action can be initiated immediately. Throughput and progress to completion statistics are also available on-line to assist management of the overall process.
8 Check Marking
Check marking can be carried out by senior markers as a quality assurance exercise. As with double marking, check marking is carried out “blind” to preserve the integrity of the quality assurance process.
9 Result Collation and Submission
After marking is complete, marks for every question of every script are available. Each candidate’s raw mark for the total script is automatically calculated. After careful and thorough system checks, the overall results are collated and submitted to the relevant examination agency electronically.