First semester academic performance: The importance of early indicators of non-engagement

Success, progression and retention of students are goals of many university strategic directions and policies. For many decades it has been recognised that the greatest focus in any retention strategy should be on first-year students. University of Otago too has goals around student success. The Strategic Plan of the institution also identified that in the context of a fiscally constrained environment, all of our activities and processes need to be assessed for efficiency and effectiveness. To this end, a pilot was undertaken in one area of the university to identify possible indicators of first-year students’ non-engagement in the first semester and their possible impact on the first semester academic performance. The findings suggest that there are indeed some indicators that predict Grade Point Average at the end of the first semester. Please cite this article as: van der Meer, J., Scott, S., & Pratt, K. (2018). First semester academic performance: The importance of early indicators of nonengagement. Student Success, 9(4), 1-12. doi: 10.5204/ssj.v9i4.652 This article has been peer reviewed and accepted for publication in Student Success. Please see the Editorial Policies under the ‘About’ section of the Journal website for further information. Student Success: A journal exploring the experiences of students in tertiary education This work is licensed under a Creative Commons Attribution 4.0 International Licence. As an open access journal, articles are free to use with proper attribution. ISSN: 2205-0795 First semester academic performance: The importance of early indicators of non-engagement 2 | Student Success, 9(4) December 2018 Introduction Student success and retention are important foci of higher education institutions worldwide. The New Zealand University in which this project was carried out is committed to enhancing student retention and academic success through research-informed approaches and evidence-based evaluation. Transition of first-year students to university was identified as an important focus for the strategic imperative of an Outstanding Student Experience in University of Otago’s Strategic Direction to 2020 (University of Otago, 2014). We saw the need to identify the key skills required for success and current gaps in the provision of support in order to address student transition. Although there is more than 40 years of research into first-year retention and student success (Braxton, Milem, & Sullivan, 2000; McInnis, James, & Hartley, 2000; McInnis, James, & McNaught, 1995; Tinto, 1982, 2000, 2007) there is no single ‘magic bullet’ intervention that has proven to address the needs of all students in any context. Retention and academic success interventions need to be tailored for particular contexts, taking into account local affordances and constraints (Nelson, Clarke, Kift, & Creagh, 2011; van der Meer, Scott, & Neha, 2010; Yorke, 2004; Zepke & Leach, 2005). In a context of considerable fiscal challenges it is understandable that universities seek to enhance the academic success and retention rate of first-year students as there is a potential reputational loss and a financial loss over the life-time of a student in an institution. Furthermore, money spent on recruiting students could be considered a ‘sunk cost’ if too many students do not stay after the first year. Interventions, therefore, that would even in a modest way enhance the retention of students after their first year, would benefit everyone. However, there is also another compelling rationale to focus on student support. For many students, leaving the university after one year of study can potentially be a negative experience that will impact on their sense of wellbeing and


Introduction
Student success and retention are important foci of higher education institutions worldwide. The New Zealand University in which this project was carried out is committed to enhancing student retention and academic success through research-informed approaches and evidence-based evaluation. Transition of first-year students to university was identified as an important focus for the strategic imperative of an Outstanding Student Experience in University of Otago's Strategic Direction to 2020 (University of Otago, 2014). We saw the need to identify the key skills required for success and current gaps in the provision of support in order to address student transition.
In a context of considerable fiscal challenges it is understandable that universities seek to enhance the academic success and retention rate of first-year students as there is a potential reputational loss and a financial loss over the life-time of a student in an institution. Furthermore, money spent on recruiting students could be considered a 'sunk cost' if too many students do not stay after the first year. Interventions, therefore, that would even in a modest way enhance the retention of students after their first year, would benefit everyone. However, there is also another compelling rationale to focus on student support. For many students, leaving the university after one year of study can potentially be a negative experience that will impact on their sense of wellbeing and self-worth.

Student support interventions
In an Australasian first-year student support context, reference is often made to different 'generations' of approaches (Kift, Nelson, & Clarke, 2010;Nelson et al., 2011). In the 'first generation' approach, teaching of generic skills for first-year students typically takes place in centrally funded learning support units. Much of this happens in one-to-one consultations and in generic workshops. In many instances, this could be considered a 'remedial' approach focused on students who are somehow perceived to be deficient or do not have the requisite skills. In the 'second generation' approach, learning support is provided through an integrated approach. In this model, development of skills is incorporated within the curriculum. Sometimes these are stand-alone discrete activities and so could still be considered as a form of generic remedial approach. Skillen, Merten, Trivett and Percy (1998) proposed extending the integrated approach to a more embedded approach where skills are integrally developed as a 'normal' way to help students to move from apprentices to experts in the discourse conventions of specific disciplines. The third generation approach is referred to as 'transition pedagogy' (Kift & Field, 2009;Kift et al., 2010;Nelson & Kift, 2009). This approach is a 'whole of university', 'whole of student', joined-up approach which may consist of academic orientation and support activities over a period of time, starting before students arrive, as well as activities embedded in the curriculum. The essence of this approach is a high level of coordination and collaboration across the whole institution, between both academic and professional staff.

Transition-focused approaches
Transition-focused programs or approaches typically focus on supporting students in becoming familiar with studying at university. They can take different formats, from extracurricular, co-curricular, to embedded approaches in programs or specific units of study. This does not mean a choice has to be made. All these three approaches can be implemented concurrently in a third generation approach.
Extra-curricular programs include academic orientation activities that are provided for students outside of their chosen study program(s). These are typically aimed at induction into the world of a university student, which can be social, academic, or both. The range and diversity of these activities are many (Kift et al., 2010;Nelson et al., 2011 A well-established and well-researched cocurricular approach is the Peer Assisted Study Sessions program -PASS (Dawson, van der Meer, Skalicky, & Cowley, 2014;Power, 2010). This particular program is focused on assisting first-year students in both academic and social transition to university. This is done through supporting students in becoming familiar with successful study habits in a social peer-led context. This program, which was developed in the U.S. in the 1970s to combat first-year attrition and is known as Supplemental Instruction, has further developed beyond the US since the 1990s where the program is generally known as PASS (Dawson et al., 2014). Research into the effectiveness of PASS suggests that in many cases participation in this program has a positive effect on both academic success and retention of first-year students (e.g., Dawson et al., 2014;van der Meer, Wass, Scott, & Kokaua, 2017).
Finally, an embedded approach is aimed at providing programs or paper-specific academic induction into the particular disciplinary context of the program or paper. This may typically involve a focus on specific academic conventions and expectations such as writing conventions and reading and note-taking approaches (van der Meer, 2012a, 2012b).

Early identification of students at risk
Because a satisfactory transition into higher education is beneficial for first-year students, it is important to identify students who may not make this satisfactory transition as early as possible. One well-researched intervention is data-based early identification of first-year students who may be at risk of failure or underperformance due to known risk factors (Dancer & Fiebig, 2004;Mallik, 2011;Mallik & Lodewijks, 2010;Nelson et al., 2011). These may include pre-entry characteristics, such as high school academic attainment or particular tertiary entry pathways (Clark, van der Meer, & van Kooten, 2008;Mallik & Lodewijks, 2010).
Other factors that could be considered risk 'flags' may indicate lack of engagement, or poor study behaviour. A 'flag' does not necessarily mean a causal reason for students' academic under-performance, but a possible indication that something may be happening that stops them from adopting good study habits. Good study habits could include class attendance, engagement with learning management systems (LMS) (Campbell & Oblinger, 2007;Newman-Ford, Lloyd, & Thomas, 2009), or in the submission of assessments. A student not displaying these behaviours can then be flagged for further attention. Embedding flags in an assessment, quiz or survey early in the first semester are some of the approaches that are recognised as effective in identifying disengagement (Thomas, 2012).

Student engagement
There has been much theorising on the factors that contribute to, or hinder, student success and retention. One of the theories is that of 'student engagement'. However, there is not a single theory, understanding, or definition of what 'student engagement' entails, its underlying factors, mechanisms or goals. Many authors have aimed to synthesise and/or summarise the literature and/or emphasise the many facets of student engagement (see Coates & McCormick, 2014;Kahu, 2013;Kahu & Nelson, 2018;Trowler, 2010;Zepke, Leach, & Butler, 2010).
Considering the many 'moving parts' in the complex 'machinery' of student success and retention, conceptualising a more holistic framework may be desirable. Such a framework might take into consideration the possible influencing personal and contextual factors, the aspects of the learning and teaching environment, co-curricular and extra-curricular affordances, and the potential goals and/or outcomes of university studies. This might then provide a comprehensive context for investigating the various aspects that contribute to student retention and success, and prompt considerations of multiple factors when considering interventions. Figure 1 provides a work-in-progress conceptual diagram of a holistic framework of student engagement. This framework is informed by many other student engagement related studies. The representation of five of the 'pillars' were inspired by the work of Kahu (Kahu, 2013;Kahu & Nelson, 2018). We build on this by separating out 'academic engagement' and adding a sixth 'pillar' that emphasises the importance of 'campus and community engagement' which reflects some of the engagement areas which are not directly related to the classroom environment but can play an important role in students' experience and achievement (e.g., Coates & McCormick, 2014;Kuh, 2003). Also, in the academic engagement pillar, we have included a 'relational engagement' aspect which recognises the importance of connectedness or relatedness as an important element of motivation and student wellbeing in the academic context (e.g., Ryan & Deci, 2000), Furthermore we have included more specific details of both the influences on and consequences of student engagement.
In this article we report on a project that sought to identify the importance of one of the aspects of the above model: early engagement as a factor in student academic achievement. Considering the proposed holistic framework, this meant we considered different aspects of 'behavioural engagement' early in the semester of the first year and investigated the impact of these aspects of engagement on one outcome, i.e. academic grade point average of the first semester. We also considered the influence of aspect of the students 'pre-entry context': their academic performance at high school and some other student characteristics (such as gender, age, citizenship status). Furthermore, we considered the possible influence of 'campus & community engagement', i.e. the possible influence of students living in a residential college, and their involvement in the PASS cocurricular program.
In summary, the research question we saw to address was: What is the influence of early engagement in the first semester of the first year on academic achievement when controlling for other factors such as, the possible influence of academic achievement at high school, living arrangements, and engagement in PASS?

Method
This project was carried out in one 'Division' of the University where retention rates have been slightly lower than in other Divisions for some years.
An invitation to participate in the pilot was sent to all academic coordinators of first-year first semester units of study in this Division. A total of 16 units of study, with a combined number of 981 first-year students (upper-year students were excluded from the analysis) accepted to be part of the pilot. Of this cohort, 62% were female, 2% international, 89% direct from high school and 67% lived in a residential college.
Early lack of engagement was assessed in the first three weeks of the semester through two proxy indicators. Firstly, through monitoring students' accessing of the institution's LMS. Secondly, monitoring students' completion of a survey that was incorporated in the LMS sites of the participating units of study. The participating units of study provided a temporary appointed retention officer access to their respective LMS sites so that she could monitor students' access of the LMS as well as enable implementation of the survey. It should be noted that we did not have control over the influence course coordinators had on students' survey completion. This may be addressed in a potential future intervention.

Figure 1. Authors' work in progress of a conceptual diagram of student engagement
Although these activities were aimed at early identification of students who were not engaging with the course, these activities also enabled the retention officer to identify students who were struggling with computer access, had technological skill deficits, or who may have had administrative issues and possibly did not have access to the LMS at that point in time.
A database was set up with data about all the first-year students enrolled in these units of study. This data set was matched with LMS data to identify who had accessed the LMS and who had completed the survey embedded in the LMS site for the participating units of study. The retention officer made contact with all students who did not access the LMS in the first three weeks. Later in the semester the retention officer also made contact with students who were referred to them by some coordinators because of lack of attendance, failure to submit an assignment or low marks on the assignment. The retention officer added data to this database about students who were contacted at different points during the first semester (progressing from email, to SMS and finally to telephone if they did not respond). The contact approach was 'passive' in the sense that the retention officer did not meet with the students face-to-face but explained the reason for their contact and/or referred them to sources of help within the university where appropriate (such as student health, learning centre, and student administration).
At the end of the semester, data from this database were matched and merged with student administration data on first semester academic performance to perform statistical analyses using SPSS-23. Table 1 shows the numbers for the various indicators we used to identify whether these indicators predicted academic performance at the end of the first semester. Some students were contacted both in the first three weeks for not accessing the LMS as well as later in the semester for other reasons (including tutorial non-attendance, non-submission of first assignment, or not having picked up their student ID card).  (n=23), tutorial nonattendance and non-submission of first assignment (n=8), low mark on the first assignment (n=6), and students who had not picked up their student ID card by week 11 (n=6). Some students were contacted for more than one reason, but they were counted once only. This ANOVA suggests that there was a significant difference between those who were contacted (M=44.20, SD=22.39) and those who were not (M=68.29, SD=15.68), F(1,976)=72.71, p<.001 with an effect size of d=1.25.

Findings
Lastly, we performed two regression analyses for the cohort who were involved in this pilot in order to see whether early engagement indicators still had an impact after controlling for different factors. The regression predictors with the first semester GPA as dependent variable included binary variables for survey completion, and contact with the retention officer, student characteristics such as domestic/international, English as a first language, gender, age, accommodation type (residential college or other), and number of PASS sessions attended. We did not include a variable for students who did not access the LMS as these students were contacted by the retention officer and therefore are included in that variable. Table 2 shows the results of two models. As not all first-year students entered university with a NCEA (New Zealand Certificate of Educational Achievement) Level 3 entry qualification, we first performed a regression analysis for all students, without the NCEA variable (N=981). In the second model we performed a regression analysis for students who entered university with NCEA (N=779) to assess the effect of academic preparedness. For this model two extra variables were entered: NCEA Level 3 results, and the decile band of the school attended (as proxy for the socioeconomic status of students).
In both models the proxies for non-engagement did indeed predict a lower first semester GPA. Controlling for NCEA weighted scores suggests that non-engaged students, as measured by the two indicators (survey completion and contact), may have been academically less strong upon entry. However, Model 2 still predicts a lower GPA of about seven marks for students who had been contacted by the retention officer, even after controlling for academic ability at point of entry.
The results for both models suggest that students who attended the Student Learning Centre-provided PASS sessions for one or more of their units of study, and students who lived in residential colleges, performed better overall to varying degrees. In Model 2 (controlling for NCEA scores), if students would have attended 12 sessions in one or more units of study papers, the model predicted a possible gain of over five marks in their GPA score in the first semester. For some academically weaker students this could mean the difference between passing the unit of study or failing.

Discussion
The results of the proxy indicators for early engagement suggest that these may be useful for early identification and early intervention strategies with students at risk of nonengagement. With regards to accessing the LMS, contacting students as early as possible may prevent students from falling behind and causing undue stress. Students who do not engage in low-threshold, low-effort and lowstakes LMS activities (such as the online survey used in this project) may also not engage in other academic activities either (for example, preparing for tutorials, completing readings etc.). Early identification of this group of students may provide opportunities for more direct engagement with these students to bring about changes to their academic study behaviours early in the semester.
In summary, the findings from the first semester suggest that lack of (early) engagement may impact on academic performance. Furthermore, the results clearly indicate that those students whom the retention officer identified as at risk and initiated contact with, were indeed those students who overall performed worse academically. A limitation of this study, however, is that only a few participating course coordinators provided the retention officer with tutorial attendance data and first assignment data.
Overall, the results confirm findings from the existing research related to the first year in higher education. Many students are overwhelmed by the transition from school to university and do not necessarily understand what is expected of them. It is important to note that lack of early engagement is a warning 'flag' and should not be seen as a function of 'laziness', or students' lack of motivation per se. Students' transition into the first year of a new educational environment can be overwhelming and produce a complex range of experiences. A more intentional, and ideally whole-ofinstitution approach to facilitating this transition may enhance more students having an equitable chance of succeeding (Kift, 2008;Kift & Field, 2009;Kift et al., 2010;Krause, 2005;Nelson et al., 2011;Nelson & Kift, 2009;van der Meer, Jansen, & Toorenbeek, 2010).
The feeling of being overwhelmed may result in students not 'getting with the program' as early as desirable, as for instance, in accessing the LMS, completing LMS-related tasks (such as quizzes, surveys), attending all teaching/learning sessions, and planning for their first assignments (Kift et al., 2010;Krause, Hartley, James, & McInnis, 2005;McInnis et al., 1995;Nelson et al., 2011;Nelson & Kift, 2009;. Being overwhelmed and the experience of information-overload in the first weeks of the first semester may also result in students not being aware of, or understanding, the support services available to them. What could not be assessed in this pilot project was whether students' academic performance would have been even worse if no contact would have been made with those were identified as being at risk. As indicated earlier, this contact was 'passive' only. A more 'active' approach, such as meeting with the students to discuss and talk through any barriers or issues and follow-up these students, might have improved their academic performance. Evidence from projects elsewhere where firstyear students in their first semester were contacted for similar reasons (Nelson, Quinn, Marrington, & Clarke, 2012) suggest beneficial results from a more active approach.
The regression results indicate that attendance of PASS sessions predicts better academic results.
Peer-assisted study sessions incorporate induction of first-year students into study approaches that benefit successful academic performance. This finding is also supported by a considerable amount of research over many years (Dawson et al., 2014;van der Meer et al., 2017). Of note is that the impact of attendance of PASS sessions remains significant, even when controlled for prior performance at high school. In other words, it is not just academically 'better' students (as measured through NCEA weighted Level 3 results) who benefit from PASS.

Possibilities for action
The following actions suggest some possibilities for enhancing first-year students' engagement, academic achievement and retention. First, monitoring students' accessing of the LMS in the first week is very important. Ideally students who do not access the LMS would be contacted within the first week, so as to minimise any possible detrimental effect of non-engagement. This contact would also send an early message to new students that the university cares about its students. This, in itself, may be beneficial to develop an early sense of belonging for these students.
Secondly, a simple short quiz or survey in the LMS site of each first year unit of study, to be completed in the first two weeks, could be an additional avenue to identify students who may have challenges in getting on board with their new academic environment. Furthermore, if the quiz is directly related to the subject matter of the paper, it could both function as a sensitising tool for students starting their engagement with new subject matter or course structures, as well as provide the course coordinator with an insight into students' perception or knowledge of some of the aspects or content of their course.
Thirdly, students' attendance, especially in small class settings such as tutorials, is known to be an important predictor of success (Credé, Roch & Kieszczynka, 2010;Mallik, 2011). Attendance monitoring in the units of study involved in this pilot varied, so it was difficult to identify all at risk students in a timely fashion. An objection that may be advanced for attendance monitoring could be the administrative burden of doing so. There are some straight-forward portable technologies to record students' attendance that could be deployed and would take little effort. Nonattendance could then prompt tutors, or other designated staff, to contact non-attendees. It would be important, however, to clearly signal to students that this monitoring is not about 'telling them off', but that the university cares. This may generate positive sentiments of belonging.
Fourthly, students' level of engagement with and/or performance on the first assessment has been found to provide another strong indication of the possible need to engage with identified students and decide on possible avenues of extra help or support, whether social, academic, health, or otherwise. Academic staff could consider making an early piece of assessment a particular focus in their 100-level units of study.
Fifthly, the difference in the academic performance of students who attended PASS sessions may suggest another intervention. This co-curricular program is one of the few academically-focussed initiatives for first-year students that has been widely researched and has been found to be effective. Ideally this program could be more widely rolled out so that all students would have the opportunity to benefit from this support initiative.
Finally, appointment of staff dedicated to early identification and support for students at risk of poor or under-performance may be worthwhile. A more active approach in supporting students and subsequent case-management could make a real difference in retention and student achievement. Furthermore, retention staff could also support teaching staff in some of the other initiatives mentioned above.

Conclusion
Overall, using a holistic framework of student engagement to conduct a study such as this was useful. Researching different aspects of student engagement as well as taking into consideration personal and contextual variables is a worthwhile effort to identify opportunities to enhance student outcomes. Using a holistic framework of student engagement could prompt researchers to consider a range of possible aspects that may be relevant and/or significant. The current project was just one modest effort to study aspects of student engagement. Other current projects will further explore additional aspects conducive to a positive student experience in their first year of university. Over time we hope to study systematically a wide range of aspects related to student engagement and evaluate which interventions may yield the greatest gains to support students' overall academic achievement and well-being, both during their time at university and beyond.