Framework for evaluating online assessment
Our framework is summarized below in Figure 1 and comprises six key design considerations and four broader contextual factors integral to evaluating and creating quality online assessment designs.
Design Considerations
1. Academic Integrity
Academic integrity addresses academic dishonesty and is concerned with the security of an assessment, including assuring against outsourcing, impersonation and other forms of inappropriate assistance.
2. Student Experience
Student assessment experiences are influenced by several factors. Positive experiences are encouraged when online assessments are convenient, maximise comfort and ease of concentration, and minimise student stress and anxiety. Assessments should be designed to reduce cognitive load and increase student motivation. Technical complications should be minimised.
3. Authenticity
Authentic assessments involve tasks that are common to professional settings, performed under conditions that are similar to those professional settings. Authentic assessments generally deal with complexity and inquiry and will optimally involve self-assessment in which students learn to evaluate and improve their workplace performances.
Well-designed authentic assessments can improve academic integrity by reducing the likelihood of some methods of cheating through unique student responses, however there is also a risk of encouraging students to cheat if the level of complexity within the assessment is poorly supported and results in cognitive overload.
4. Information Integrity
Information integrity refers to the protection of student personal information and data. Well-designed online assessments will minimise the risk of unauthorised access to student personal details such as demographic and biometric data, and content that students have generated during the assessment.
Tension may arise between academic integrity and privacy/security of student information in online assessments, for example some technology (e.g., artificial intelligence used to assure online exam invigilation) requires provision of personal data and identifying information that may be susceptible to security breaches.
5. Quality Feedback
High quality assessment feedback is timely and supports students’ understanding of their performance and how to improve in the future. Good feedback practice will encourage formative dialogue between students, their peers and their educators, as well as scaffold students’ capacity for self-assessment. Online assessment may be further improved with support for feedback in multiple formats (e.g., annotated, audio or video feedback), and with automation techniques for improved scalability.
6. Equity of Access
Equity of access involves removal of all barriers for students’ completion of the online assessment. This enables assessment conditions to be customised to meet individual student needs, and affords both technical and logistical accessibility for all students including provision of live technical support. Potential bias and discrimination also needs to be avoided where automated invigilation, grading or feedback are used.
Broader Contextual Factors
Our original framework included scale of delivery and resource limitations as broader and interrelated contextual factors that influence decisions about assessment design. Through our research we found support for two additional contextual factors, institutional policies and accreditation requirements.
1. Scale of Delivery
Running online assessments with large cohorts requires dedicated attention to improving student experience while reducing teacher time commitments and resourcing. This involves adopting techniques that expedite the collection of assessment data, grading, provision of feedback, and assessment management and administration.
2. Resource Limitations
Resource limitations that impact the design, implementation and marking of online assessments primarily involve the available staff time and budget but may also include provision of physical resources such as technologies and spaces. Educators may adopt techniques to reduce resource requirements, such as recycling assessment strategies and materials or automating some aspects of marking and feedback.
Resources limitations are related to issues of scalability, as there may be increased resource requirements for the introduction of a new form of online assessment within a large cohort relative to a smaller cohort.
3. Institutional Policies
Institutional policies relating to areas such as curriculum design, academic integrity, equity, diversity, access, and technology will shape how online assessments should be designed and adopted. Additionally, the selection and implementation of online assessments may be influenced by the assessment culture in the discipline, unit or faculty.
4. Accreditation Requirements
Programs that are accredited by professional associations or regulatory bodies often have additional conditions on assessment design. For example, these may specify requirements for verifying each students’ identity when completing the assessment, or the extent to which the assessments need to directly evaluate students’ performances of workplace tasks.