Benchmarking Australian Enabling Programs for a National Framework of Standards. A Practice Report

Abstract


Introduction
Enabling programs in Australia provide non-traditional students an entry pathway into university, equipping them with the skills to access and participate successfully in undergraduate study (Syme, Roche et al., 2021). Enabling programs have long been part of the widening participation agenda (Bradley et al., 2008) with the aim to disrupt educational disadvantage with access to higher education . Nationally, enabling programs attracted 32,579 student enrolments in 2020 Enabling education programs in Australia assist students, who would otherwise have been excluded from higher education, to transition into undergraduate study. These programs emerged independently in response to the needs of individual universities and the varying cohorts of students they serve. The exclusion of these programs from the Australian Qualifications Framework (AQF) has meant they remain unregulated, with no national framework for standards. The development of academic standards is a dynamic, consensus driven process, and benchmarking provides a method through which academics from across institutions can work in partnership to reach shared understandings and improve and align practices. This practice report outlines the results of the first comprehensive cross-institutional benchmarking project involving nine Australian universities and demonstrates there is shared understanding of the standards of enabling programs between institutions. These findings will contribute to the establishment of national standards for enabling programs in Australia.
(Department of Education, 2022), representing a fourfold increase across the sector since 2001 and a significant widening participation strategy. The students are typically First-in-Family, Indigenous, and from equity groups such as low socioeconomic, or rural and regional backgrounds who have often experienced disadvantage in their schooling and would be otherwise excluded from higher education (Jarvis, 2021;Pitman et al., 2016). Enabling programs provide the opportunity for non-traditional students to learn skills to navigate the academic conventions and expectations of higher education and transition successfully into undergraduate study (Syme, Roche et al., 2021).
Enabling programs are not included in the Australian Qualifications Framework (AQF), having emerged as non-award preparatory programs at individual universities across Australia to provide access for local cohorts seeking alternative pathways into higher education. There is no national framework to enable transparency and to compare learning outcomes and standards of achievement across the sector. This lack of quality assurance means that students can be denied portability of qualifications between institutions (Pitman et al., 2016;Shah & Whannell, 2017). As well, enabling educators often work in isolation at the periphery of their institutions (Crank, 2022) with limited opportunities to demonstrate the quality of their programs through external peer review, scholarship and research (Bennett et al., 2017). As such, enabling programs can be perceived problematically as diverse and disparate (Baker & Irwin, 2015). Nevertheless, some studies have drawn comparisons between programs. Hodges et al. (2013) described some challenges across five enabling programs, particularly for student retention, and made several recommendations including the importance of low stakes, early assessments, and the establishment of an enabling education community of practice. Relf et al. (2017) established that common principles underpinned curriculum design across three enabling programs. The first cross institutional benchmarking of enabling programs by Syme, Davis et al. (2021), which compared three programs, established the comparability of learning outcomes and standards of achievement in three subjects: study preparation, academic communication and fundamental mathematics. This was significant given that the programs at each institution were developed in isolation in response to the needs of students in their footprint. Drawing on the commonalities established in the work of Relf et al., (2017) and Syme, Davis et al. (2021), the National Association of Enabling Educators of Australia (NAEEA) (2019) published national learning outcomes to inform the development of a national framework for enabling education.
The Tertiary Education Quality Standards Association (TEQSA) through the Higher Education Standards Framework (Threshold Standards) requires higher education courses to be periodically formally reviewed, including undertaking external benchmarking activities (Australian Government, 2021). Such reviews would typically examine course design and content, learning outcomes and students' achievement of these, and assessment practices. Benchmarking assures quality, informs internal improvements, and fosters collaborative sector development (TEQSA, 2019). Sadler (2017) argues that, while quality assurance measures in higher education often focus on inputs such as the university facilities, policies and processes, academic quality is in fact a measure of what students can do and to what standard. Benchmarking of assessment tasks, student responses and grading provides a framework for determining output quality (Sadler, 2017;Sharp, 2017). Further, it engages academic staff in the establishment and monitoring of consensus driven standards within the context of collegial and robust dialogue through which often implicit understandings of academic standards are made explicit (James, 2003). The open and genuine sharing of teaching and assessment practices through benchmarking also diminishes the risk that standards become prescriptive and homogenising, inhibiting rather than fostering the innovation (James, 2003) that is central to improving teaching, learning and student success. The benchmarking framework employed in this study provides a rigorous but practical method that could be applied more broadly across higher education programs in alignment with the Higher Education Standards Framework's benchmarking requirements for assuring program quality.
This practice report encapsulates the findings of the first large scale benchmarking study of nine Australian enabling programs, selected for their varying sizes, structures and geographical reach, yet accounting for approximately half of enabling program enrolments nationally (Department of Education, 2022). The study, which builds on a previous study by Syme, Davis et al., (2021), gains reliability from an established benchmarking framework and templates to compare the learning outcomes, curriculum, and standards of achievement in three key subjects from each of these nine programs. Key aims of the benchmarking process were to see if the original findings were applicable across a wider number of programs with the aim of establishing a national framework of standards for enabling programs, thereby providing quality assurance, transparency and potentially, portability of qualifications for students. A further aim was to test the benchmarking framework and templates for applicability across a wider scale. Finally, the project aimed to provide an opportunity for enabling educators to collaborate and share best practices for the purposes of continuous improvement and to generate opportunities for cross institutional scholarship.
The nine participating universities in this benchmarking project are provided in Table 1. While the length of programs varied from three to eight subjects, all were characterised by minimal entry requirements and included core and elective subjects.

Methodology
The project adapted Morgan and Taylor's (2013) benchmarking framework, which outlines the processes required to establish benchmarking protocols, recruit benchmarking partners, develop an evidence portfolio, compare assessment practices, investigate standards, and analyse and report findings. The structure employed by the project allowed for consistent and efficient collection and comparison of curriculum and assessment data from across nine different enabling programs, and included an additional phase which emphasised improvements in practice. These phases included: 1. Prepare for benchmarking by recruiting university partners and agreeing to the scope and terms of the project. 2. Develop an evidence portfolio of curriculum documents, assessment tasks, blind assessment scripts and student evaluation data. 3. Compare assessment standards through collaboration and discussion of processes and outcomes including blind marking and moderation practices as well as learning outcomes and curriculum comparison, using consistent templates. 4. Analyse and discuss findings across programs by comparing the data in the templates. 5. Improve practices through identifying areas of best practice and future improvements. 6. Final reporting through collation of findings into a report with recommendations to NAEEA on the approach for a national framework of standards.
This methodology, replicated from the original benchmarking study (Syme, Davis et al., 2021), was underpinned by the researchers' commitment to engage in the "rigorous, frank and confidential exchange of information" (Taylor & Morgan, 2011), reflected in a formal cross-institutional agreement and in ethics approvals (Phase 1) (p. 5). The nine universities divided into groups of three, each group led by a member of the original project (Syme, Davis et al., 2021) to develop an evidence portfolio (Phase 2), drawn from the first program session for 2021 at each university. Robust discussions took place over a year-long process of online fortnightly meetings at which the nine researchers compared learning outcomes, topics, and assessments in the same subjects as the original project in study preparation, academic communication and fundamental mathematics. This information was recorded in templates in a shared repository for easy access and comparison. The overall structure, delivery and enrolment numbers of programs were also documented. The learning outcomes for each subject were then mapped against the NAEEA's Common Learning Outcomes for enabling programs (NAEEA, 2019). Each group investigated moderation practices and compared assessment standards through a blind marking process as part of Phase 3. Four de-identified scripts of the final written assessment for each subject were distributed amongst the marking team at each institution within each group of three, along with the assessment instructions and marking rubric/criteria provided to students. The marks and grades were entered into the relevant template. Each group then analysed the grades and marking process and reported on the findings to the group of nine where further analysis took place (Phase 4). In line with the tenets of best benchmarking practice (TEQSA, 2019), a series of shared good practices was revealed as well as areas that needed improvement (Phase 5), informing the ongoing development of programs to improve student outcomes. The final report was drafted and distributed for feedback and editorial input (Phase 6). The group of nine presented an overview of the report to the NAEEA National Conference in Adelaide, Australia 2022 . The final report was then uploaded to the NAEEA website (https://enablingeducators.org/).

Comparable Curriculum
All nine programs included a core academic literacy subject, with each found to have almost identical learning outcomes, topics, content and assessment. These subjects focused on explicit teaching of the culture and practices of the university environment, research and critical reading, and communicating ideas using the language and conventions of a written research essay. All subjects included scaffolded assessments, such as responses to readings, essay planning and drafting, which culminated in the completion of a final essay. Each allowed for iterative feedback and feedforward processes to support the development of student writing and to build students' confidence in attempting to write an academic essay. Rubrics that facilitated tutors' marking consistency and allowed for transparency of the grading process for students were also a feature shared across all subjects. These findings reflect a common understanding of the academic literacy demands required of students entering university, the standards expected, and the need to develop students' abilities to employ the language and academic conventions of the university context.
A study preparation subject was offered at six of the participating universities, while two offered a comparable subject that combined university study preparation and academic literacy topics, and one offered a week of intensive study preparation in the first week of semester. The benchmarked study preparation subjects were found to have very similar learning outcomes, topics, content and assessments. The subjects included explicit teaching of study strategies and focused on equipping students with an understanding of academic culture, expectations, and conventions. Activities and assessments provided opportunities for students to reflect on their learning experiences to build resilience. This shared approach emphasised a demystifying of university discourses and practices and empowering novice students to navigate the university environment.
Benchmarking of the third subject, fundamental mathematics, also revealed a high level of comparability. Although there was some variation in content, the learning outcomes and topics were very similar. There was an emphasis on real-life problemsolving activities and assessments, with only one subject not identifying this expectation explicitly in its learning outcomes. Approaches to assessment varied and there were generally more assessment tasks than in the study preparation and academic literacy subjects. Weekly engagement activities, quizzes, exams and written problem-solving assignments made up these tasks. The requirement for students to undertake more advanced mathematics study for entry into specific mathematics or engineering related undergraduate programs was addressed in five programs through further mathematics electives. Two required elevated levels of achievement in the fundamental mathematics subject, and two required students to enter a pathway degree. Nevertheless, the comparable curriculum across the benchmarked fundamental mathematics subjects indicates a widely shared understanding of the mathematical abilities expected of students exiting these subjects and entering undergraduate study.

Alignment with NAEEA Common Learning Outcomes
While there is no external framework to which enabling programs are formally aligned, the NAEEA Common Learning Outcomes describe the program level outcomes which students exiting from an enabling program could be expected to have demonstrated. These include a knowledge of the university context, cognitive, information, communication, academic literacy and learning skills, and the application of knowledge and skills through engagement, independent learning, critical thinking and appropriate student practices. Six of the nine programs had explicit statements of program level learning outcomes, and these were compared with the NAEEA Common Learning Outcomes. This comparison distinguished whether these learning outcomes were explicit or implied in program overview documents (Figure 1).

Mapping of Program Level and NAEEA Common Learning Outcomes
The learning outcomes for the three benchmarked subjects from each university were also compared to the program learning outcomes for enabling programs identified by NAEEA (2019). In each case, the curriculum documents included explicit and accessible statements of learning expectations that aligned closely with the Common Learning Outcomes. For each program the three subjects together addressed these common learning outcomes, which were often embedded across multiple subjects within the programs. This indicates a high degree of consensus regarding the essential knowledge, skills and attributes expected of students exiting an enabling program. There is scope for each university to review its learning outcomes during accreditation and reaccreditation phases, both at program and subject levels, to consistently and explicitly articulate and align with the NAEEA Common Learning Outcomes. These findings further support the establishment of a national framework for enabling education programs based on the NAEEA Common Learning Outcomes.

Comparability of Standards
Blind marking of four de-identified student scripts from each of the three subjects across the nine universities revealed a high degree of comparability of standards expected of students across these subjects. Markers undertook this task without any premarking moderation discussions, drawing on their existing knowledge and experience to interpret the task descriptions and rubrics provided and to make grading decisions. A total of 100 student scripts, including 28 from the eight benchmarked study preparation subjects, and 36 each from the academic literacy and mathematics subjects were blind marked by three markers from different universities. A total of 55 scripts received identical marks from all three markers, while 34 varied by one grade, and 11 varied by more than one grade. This activity led to considerable discussion about the practices of each university, particularly in identifying quality practices such as the use of assessment task descriptors which include clearly explained and contextualised task requirements, along with clear and specific guidelines for students, and the use of rubrics which are clear, written in accessible language and lend themselves to shared interpretations among both students and markers. While mathematics marking was somewhat more closely aligned than the other two subjects, this nevertheless reflects a high degree of consensus regarding the standards expected of students across all these subjects.

Moderation Practices
Discussions revealed all programs included a moderation process to ensure markers have a shared understanding of the assessment requirements, marking criteria and standards. Through benchmarking, good moderation practices were identified, including having a process for multiple markers to compare sample marking outcomes prior to undertaking assessment marking, using a checking mechanism to ensure consistency before grades are released, and presenting marking in such a way

Common Learning Outcomes
Explicit Implied that students can interpret their marks and feedback and use this to feedforward into their future work on assessment tasks. These practices help to address the challenge of achieving consistency of marking across teaching teams within subjects.

Student Satisfaction
Student feedback at the end of each teaching session, usually in the form of a Likert scale rating, measured student satisfaction with their subjects. From a possible score of five, student evaluations ranged from 4.0 to 4.9, which were generally higher than average university wide student evaluations. This suggests that students value the learning in their enabling subjects and attests to the quality of teaching practices within enabling programs. Benchmarking discussions identified student-centred learning and teaching practices that are meaningful to students' academic futures, along with approachable, knowledgeable and skilled lecturers and tutors, a pedagogy of care and a strengths-based curriculum as good practices which contribute to high levels of student satisfaction for students transitioning into tertiary study.

Impacts of Benchmarking
While these results demonstrating the comparability of curriculum, assessment practices and standards are specific to enabling programs, a further finding of the project was the high value nature of benchmarking with several positive impacts emerging. The collaboration across universities and sharing of learning and teaching resources has led to some changes to existing subject content, assessment and pedagogies to improve student outcomes. These include: • the introduction of a focused study preparation subject in programs that did not previously have one • reducing the number of assessments in some subjects to allow students time to focus on and benefit from feedforward • further embedding types of assessment which allow for better scaffolding • developing clearer assessment tasks and rubrics to better align with the learning outcomes and use consistent language to facilitate common interpretations • co-creating shared understandings of assessment design and rubrics and increasing the transparency of standards and assessment practices • reviewing of assessment approaches including use of project-based models and real life related problem-solving tasks alongside quizzes and exams in mathematics • redesigning of subject content with a focus on alignment with future disciplines • sharing of effective moderation practices to improve consistency Significantly, benchmarking can facilitate the establishment of communities of practice to provide a powerful platform for genuine sharing to improve practice, a means of validating good practice, and an avenue for recognising and valuing the work of educators.

Discussion
A key finding of this benchmarking project is the high level of comparability in the learning outcomes, curriculum, assessment and standards of achievement in the three benchmarked subjects across the nine programs. This reflects shared understandings of quality curriculum and assessment practices for supporting students transitioning into university, and broad consensus about the knowledge, skills and attributes students were expected to demonstrate upon exiting their enabling program. Each emphasised familiarising students with the environments, discourses and expectations of university study. While valuing the diversity of programs which continue to serve varied cohorts, this finding provides assurance of the quality and validity of programs in preparing non-traditional, novice students for university study. This is particularly important as it demonstrates there is consensus regarding the standards expected of exiting students despite the current lack of national standards or inclusion in the AQF.
The study provides compelling evidence to support the adoption of the NAEEA program learning outcomes as the national standard for enabling education across Australia. It also provides a coherent position from which the NAEEA can continue to engage with the current Australian Universities Accord process (Australian Government, 2023) to include a standard for enabling programs. A national framework of standards is critical to providing completing enabling students with portable qualifications and securing the recognition of the instrumental role enabling education plays in preparing non-traditional, novice students for the rigours of higher education. It allows for the coherent communication of the quality of enabling programs, assuring the leaders and sponsors of discipline-based programs of a pipeline of well-prepared students. Improving access to higher education for students who would otherwise be excluded increases the capacity of the Australian higher education sector to address widening participation agendas and to qualify the numbers of tertiary graduates required of modern economies.
The project provided the opportunity for enabling educators to meet and compare their programs in a spirit of collegiality for continuous improvement to the benefit of students, staff and universities. The establishment of an NAEEA Community of Practice, which to date has, significantly, engaged 75 academic staff from across Australia, is another key outcome, ensuring opportunities for ongoing collaborative scholarship for enabling educators. Cross institutional opportunities for scholarship are crucial to showcase the transformative role enabling education plays in supporting novice, non-traditional students to transition successfully into higher education. Grounding enabling education in scholarship is particularly important to gain visibility and recognition for enabling educators' work which is often at the periphery of universities (Baker et al., 2022).
Although the broad range of programs included in this project developed in different parts of Australia to meet the diverse needs of their student cohorts, the project identified many shared good practices including the clear alignment of learning outcomes, curriculum and assessments, scaffolding of assessments, clear and accessible rubrics and the importance of moderation to ensure consistency of marking outcomes that align with enabling and transition pedagogies (Bennett et al., 2017;Hattam et al., 2022). These shared good practices offer much for the broader university teaching community as they encapsulate principles of "good" teaching that benefit all students and their lecturers (Chickering & Gamson, 1987) and enhance student retention and success.
The benchmarking project demonstrated the adaption of Morgan and Taylor's (2013) framework proved an effective tool to compare the learning outcomes, curriculum, assessments and standards of achievement. The templates, included in the published report , provided a realistic and workable structure for the consistent collection, collation and analysis of data, and supported the coherent reporting of findings. This benchmarking framework has potential to be used across the enabling education sector for ongoing benchmarking and external review, something that has not been readily available to enabling educators to date. The broad reach of the project has provided a platform to promote and support a culture of ongoing peer review among enabling programs, supporting continued improvement of student retention and success. It also provides an accessible external benchmarking method that could be adapted for programs across the higher education sector.

Conclusion
This benchmarking project has contributed to a more coherent and comprehensive sector wide picture of enabling education curriculum, assessment practices and standards of student achievement. Based on the finding that programs and subjects have comparable learning outcomes, and that standards across these enabling programs are also comparable, there is impetus to develop a consistent method for tracking enabling education students' achievement and retention following their transition into undergraduate study. Such a sector wide approach will support an evaluation of the impact of enabling education and its role in widening participation in higher education at a national level. The next stage of the project will be to document the success and retention of enabling students completing their first year of undergraduate study across the sector. The collation and analysis of sector wide evidence underpins the ongoing provision of enabling education for future generations of students by making visible the critical role these programs play in widening participation in Australian higher education to governments, institutions, and communities.