Does Course Specific Nudging Enhance Student Engagement, Experience and Success?: A Data- Driven Longitudinal Tale

Low levels of online student engagement impact negatively on student success and adversely affect attrition. Course learning analytics data (CLAD), combined with nudging initiatives, have emerged as strategies for engaging online students. This paper presents a mixed method case study involving a staged intervention strategy focussing on the employment of timely, strategic communication interventions conducted across 19 courses and six disciplines. The research methodology utilised CLAD, online surveys, student interviews and student evaluations of teaching. The findings substantiate benefits for both academics and students. Academics benefitted from the provision of a relatively simple, accessible and proactive intervention for increasing students’ capacities to be more in control and engaged in their learning. Students benefitted as the intervention accentuated critical resources to assist them to better address assessment requirements, align their expectations more realistically with those of the course, and more readily demonstrate their learning obligations and responsibilities.

The literature shows that LAD are most often utilised to implement and evaluate faculty and institutional interventions designed to identify and support higher education students at risk (Nelson, et al., 2009). Communication initiatives, or nudges, have also been used in conjunction with LAD at institutional levels. Nelson et al. (2012), for example, found that LAD, in combination with nudges, can be effectively applied to a range of learning contexts and student enrolment situations and that their impact on student persistence could be sustained for 12 months. However, while LAD are increasingly used to track student engagement at institutional levels, harnessing LAD to effectively and efficiently support student engagement in specific courses is less well researched. Stone, (2016) and Lawrence et al., (2019), for example, argue that while LAD are increasingly used to track student engagement, uncertainty remains about how to harness the data effectively and efficiently and analyse them accurately to support students.
There have been initiatives in pedagogical and course specific contexts however, with course learning analytics data (CLAD) supported by nudging interventions to enhance student engagement. Nudges are intentional, timely and strategic communication strategies focussed on motivating students to engage with critical resources and activities (Selinger & Whyte, 2011). Damgaard and Nielsen (2018), in their meta-analysis, reviewed the positive impact of nudges in pedagogical contexts, citing their capacities: as prompts (Kizilcec et al., 2014); as exemplars of the work of high performing students; as reminders to students of their previously stated study goals and as software which automatically blocks distracting websites once students exceed a certain limit (Patterson, 2018); as strategies to provide personalised information about students' abilities or effort levels (Pistolesi, 2017); as social belonging interventions (Broda et al., 2018); and as 'strengths-based, educative' nudges about benefits gained from a resource/activity which were more persuasive than adopting a deficit approach that nags students about not having engaged with them (Benarzi et al., 2017). Buchs, et al. (2016) utilised nudges in specific subjects to explain to students why and how to cooperate as part of a learning task.
The project presented in this article extends an earlier project at USQ (see Lawrence et al., 2019) which combined CLAD with nudging in three courses in the disciplines of Nursing, Education and Engineering, involving 892 student participants. Only the low/non engaged students were nudged with the findings illuminating the benefits for students by alerting them to critical resources for their assessment requirements. Academics benefitted in terms of harnessing and integrating CLA to inform strategic and targeted approaches to integrating course specific nudges. Overall, the 2018 project concluded, course specific nudge interventions using CLAD represented a proactive (and relatively simple) approach that enabled students and academics to share and fine-tune their ways of working in courses to purposefully engage students.

Project Rationale
An expanded project team extended the scope and practice of the 2018 project by a) upscaling the number of course examiners (CE) and courses engaged in the project, and b) incorporating additional steps in the nudging intervention to further address the problem of the low levels of online engagement exhibited by students in course LMSs, and by so doing enhance the quality of the student experience and student success. By utilising CLAD and nudging, the new project aimed to make course expectations and requirements explicit to students in a just in time, just for you intervention as well as to assist academics to recognise the key role played by student engagement and timely nudges. In the longer term, the project team also considered that the potential existed for the amplification of the academic culture in terms of the University's primary course delivery tool.

Project Methodology
The specific objectives of the intervention were to investigate how CLAD and nudges could be used to increase early engagement (Weeks 0-5) by providing explicit guidance to students about which course resources/activities were important to access and by then connecting with non/low engaged learners to encourage their engagement with these important course resources. The research investigation was to explore whether the use of timely, strategic and encouraging communications would increase to students' engagement with critical course resources/activities.
The nudge intervention was implemented in semesters 1 (S1) and 2 (S2), 2019. The aim of the S1 implementation was to observe its impact and then refine the intervention based on those observations in order to intervene in S2. In S1, the intervention was implemented in eight courses spanning four disciplines (n=1176) (806 were enrolled in a nursing course) with 95 students receiving a nudge. In S2, the intervention was implemented in 10 courses (n=477) with 65 nudges sent to students who had not engaged with key course resources. Six of the courses were offered in both S1 and S2, and the intervention was implemented in both semesters in these courses. Overall, the seven CEs, comprising the project team, contributed 19 courses in six disciplines (Education, Physics, Accounting, Science, Nursing, and Town Planning) across two Faculties (Health, Engineering and Sciences (HES) and Business, Education, Law and Arts (BELA).
The project encompassed a mixed method case study to develop, refine and strengthen early staged course-intervention strategies to encourage online engagement using both promotions (prompts) and nudges to raise awareness of key resources. A three-stage process for promotion and nudging was followed with the key steps for the CEs encompassing: • Identifying five key course resources/activities in the first five weeks of the semester and determining when student engagement with these resources was most important; • Developing a strategic nudge plan to fine-tune nudging timing and intent, including: o A prompt, a communication provided to an entire course cohort and posted, for example, as a NEWS announcement or message on the learning management system (LMS) noticeboard to highlight to learners a key resource or activity to focus on for a specified week; o One week after a key resource had been promoted, a targeted, personalised, encouraging communication (a nudge) (either a private LMS message, email or text), using CLAD and sent only to those students who had not yet accessed that resource. The nudge emphasised the importance of a resource to their assessment, or course objectives. • The research team engaged in weekly critical reflections, reflecting about ways to better assess how/when/ what to nudge, develop nudging templates, monitor challenges and adopt continuous improvement strategies.

Data Collection
Student data was collected through CLAD, pre and post-intervention online surveys, voluntary post-study online interviews, and student evaluation data.
• Learning analytics data. To gauge the impact that the nudge intervention had on online student engagement, the team recorded their nudge-related engagement statistics for both semesters of 2019. This was achieved by using CLAD to record how many students had engaged with specific course resources before they were nudged to access that resource and then how many students had accessed it after the nudge had been given. • Online survey. Students voluntarily participated in a pre and post-study survey to determine the perceived impact of these nudges. The survey was conducted in both semesters and administered online using the LimeSurvey platform. Students received a URL link to the survey as part of their communication, and were directed to the home screen and landing page. The survey was fully anonymous with no chance of re-identification. In the survey qualitative questions asked students to comment on how helpful the nudge intervention was for their learning in the course they were enrolled. The post-intervention online survey was emailed to all students at the end of the semester and 58 responses were received across cohorts. The survey contained a mixture of closed-ended and open-ended questions. The closedended questions asked students to evaluate, on a Likert-scale, certain aspects of the nudges. Descriptive statistics were used to summarise student responses to the closed-ended questions. • Student interviews. Student interviews were instigated to further explore the views of students about online engagement and their experiences with the nudging. This data was collected in S2 only as a way to further explore students' perceptions of the interventions. Participants (students) (approximately three from each course, across 13 courses targeted for the study) were approached via email by a non-teaching team member and asked to participate in a research interview. A total of 9 students were interviewed in a 30-minute online interview. The focus of the interview was to gain an insight of the participants' perceptions of the intervention, that is, to examine the impact of the interventions and explore the use and effectiveness of the 'nudge' communication interventions for enhancing online engagement. The students were approached by a non-teaching research team member and transcribed and anonymised by a third party so there was limited/no conflict of interest. • Student evaluations of teaching (SET). The team also collected any SET comments related to the use of nudges in their courses.

Data Analysis
The quantitative analysis of data included click analysis of the CLA for each course representing students' engagement behaviour with critical course resources, quantitative analysis of the surveys to determine the efficacy of the intervention in terms of "change of engagement behaviour" with key resources. A 5-point Likert scale was used with the median value presented in the results.
The interview data were transcribed verbatim by a professional transcription service and analysed using both NVivo and manual thematic coding. The interview transcripts were coded using content analysis techniques where the data was analysed through a process of induction then the researcher constructed and reconstructed meaning in the data in relation to the research question (Janesick, 2000).
To ensure and increase data dependability, further manual qualitative analysis of the student perspectives on the intervention, gained through interviews, surveys and the SET data, was conducted. Criteria for qualitative enquiry: credibility (multiple sources of data), transferability (purposive sampling, interview schedule), dependability (presentation of frequencies alongside quotes in the interview results, independent coding by two researchers or from two analysis methods [e.g., manual and computer]) and confirmability (presentation of the participants' own voices) (Decrop, 2004) were adhered to in the research and analysis of the qualitative data.

Findings
To gauge the impact that the nudge intervention had on online student engagement CEs recorded their nudge-related engagement statistics for both semesters. For most of the course resources that were promoted and nudged, there was an increase in student engagement; and encouragingly, in S2 after the approach had been refined, there were quite large increases in engagement following a nudge. The results illustrated there was an increase of more than 5% in student engagement after the nudge for most courses and that for some courses there was an increase of greater than 20% after the nudge. Table 1 illustrates the impact of the nudge intervention in terms of the average increase in the percentage of students who accessed the nudged resources after a nudge was provided. * Increase is derived by subtracting the percentage of the course cohort who had accessed the key resource before the nudge was given (recorded on the day the nudge is given, just before it is sent) from the percentage of students who had accessed the resource one week after the nudge was given. Figure 1 below depicts the pre-nudge statistics representing the percentage of students who had accessed a critical resource before having received a nudge reminding them of the importance of accessing that resource; the post-nudge statistics represent the percentage of students who had accessed that resource after they received that nudge. Pre-nudge data was collected on the day the nudge was given (before it was given); post-nudge data was collected one week after the nudge was given. A comparison of these graphs shows that in S2, after the nudge approach had been refined, there were larger increases in the percentage of students engaging with a resource after the nudge was given. These larger increases provide evidence for the potential effectiveness of the nudge intervention when implemented in a more targeted, planned manner.

Figure 1
Comparison of Effects of the Nudges in Semesters 1 and 2, 2019 Figure 2 presents a boxplot depicting a selection of five courses' percentage change in engagement in both S1 and S2 from all disciplines (EDM8012 and EST4100 are education courses; URP3201 is a Town Planning course). Three of these courses were also offered in S2. The boxplots from S1 and S2 suggest an increase in student engagement. If we compare the means, these are marked "x" within the boxes. For example, EDM8012 S1 had nine nudges and the mean percentage increase in engagement was 4.5%. In S2, for the same course with 11 nudges, the mean increased to 21.3%. If we compare the middle 50%, these are the boxes (or rectangles in each of the courses). For example, in ACC2113 in S1 (11 nudges) middle 50% range is from 2.9% to 5%. This is much lower than S2 (eight nudges) where the middle 50% range is from 8.53% to 18.1%. The result is not only more engagement overall, but the nudges seem to be affecting more of the student engagement in resources.

Student Surveys
Pre and post-intervention surveys were used to gauge students' perceptions of the usefulness of the intervention with students asked to answer a mix of qualitative and quantitative questions. Quantitative questions asked students to indicate on a 5-point Likert scale (where 1 = not helpful; 2 = slightly helpful; 3 = moderately helpful; 4 = helpful; and 5 = very helpful) how helpful they found the nudges generally (average Likert response rating was 3.8, n=38). It also asked students to consider how helpful they found different types of nudges, including: prompts early in the semester with study suggestions and tips (average Likert response rating 3.9); early prompts and communication by teaching staff to access key resources (4.1); prompts encouraging participation in forums (3.6); prompts encouraging participation in an online zoom tutorial or lecture (3.6); tips for addressing assessment pieces (4.4); reminders about key weekly tasks and activities on which to focus (3.9); and prompts that shared % of students who had already engaged with key resources to 'motivate your engagement ' (3.4). These responses showed that students were most receptive to nudges that prompted them to access key resources or gave them information about the value of the resource in supporting their assessment pieces.
Thus, in the surveys the student feedback collected was largely positive, showing the intervention was useful for keeping students focused, providing a sense of support and reminding students to access key resources. Negative comments related to the tone, length, or frequency of receiving multiple nudges (some students receiving a nudge for each of the separate critical resources in the same week) and showed that some students felt there was too many nudges from the educators (referred to as over-nudging), or the tone of the communication was too "big brotherish" and thus lead to increased feelings of stress. Overnudging, or where the language adopted for a nudge was discouraging, punitive or coercive, was referred to as a "nag" (see Lawrence et al., 2019).

Student Interviews
Qualitative data obtained from the student interviews reinforced that the nudges were a source of motivation for them, prompting them to recognise the value and importance of key resources. As one student commented in the interview, "they do remind you to go … and look at that". Nudges were also perceived by students to be evidence of online teacher support and caring for students in relation to their success and learning journey and as part of the teacher/learner relationship. For example a student commented that, "I actually think that when a teacher shows interest, it spurs me on to want to do well and keep going even when it feels like it is too hard to keep up".
Another student reflected, "at first I felt surprised to be noticed, but it also made me feel like the teacher was wanting me to stay on track, so I then felt supported". Project findings verify that the intervention assists students to develop self-efficacy. For example, one student noted, "I did receive a message … it worked in getting me to catch up!" An important benefit that emerged from the interviews was that the intervention assisted students to organise themselves more competently, overcoming a repercussion of online learning which You (2016) identified as the failure to study systematically. Lawrence et al., (2019, citing Cerezo et al., 2016 report that online learning requires more effort when deciding what, how, and how much to learn; how much time to invest; when to abandon or change learning strategies; and when to increase effort. You (2016) also noted that the autonomous nature of online learning means that students need to be more responsible for their own learning.
From the student perspective gleaned from the interviews overall, the nudges were largely successful in increasing student access to key online resources; and on the whole, students seemed to respond favourably to the nudges. Nudges provided students with a source of motivation prompting purposeful engagement and a reason to engage with a resource, as well as a "perception of support" from the online teachers.

Discussion
As well as highlighting the increase in student engagement that resulted from the nudges, the findings also illustrate that the increases in student engagement statistics were much higher in S2 after the intervention had been refined and the team's approach to nudging changed following their S1 experiences. It was noted that not only did many more of the nudges in S2 show increases in student engagement, but the percentage increases were also considerably larger. While these results are rewarding for the team, further investigation is needed to determine whether this increase in engagement was due just to the nudges or a combination of the nudges and a number of other factors. For example, what was the engagement compared with other similar resources that were not nudged?
One of the refinements for the S2 nudge intervention emerged from the project team's S1 reflections, as well as observations from the S1 student engagement data. These were considerations about what resources to promote and how many to nudge. For example, the project team noted that while an initial nudge on a key resource was often effective in promoting increased student engagement, any subsequent nudges of the same resource led to only minimal increases in engagement. While this is expected, over time the subsequent nudges had less affect and in turn, led to 'engagement fatigue' thus suggesting there is an interaction between number of resources to nudge and number of nudges per resource. Given this, the S2 approach was more targeted and conservative about the promotion of resources (no more than five-seven overall for the first five weeks of semester) and about the subsequent nudging of students regarding these resources (only once, unless the course examiner felt there was a need for a second nudge).
A number of learning outcomes emerged from the CE's reflections. These equated to capacity building and included the outcomes that indicate: • Well-timed and well-crafted nudges written for the specific students in the courses can increase engagement without being a "nag" to students. Evidence of this occurred at the undergraduate (1 st and 2 nd year) and postgraduate levels, through increased activity on the course LMS and also (indirectly) improved SET Survey outcomes.
• Nudging works if it is applied appropriatelyit is therefore important to have a plan or strategy for implementing a nudge intervention. This strategy should plan what to nudge, when to nudge, and who to nudge, and how to nudge (i.e. the wording to be used). The project team developed a "how-to" guide to encourage academics to use nudging in their courses' curriculum design (to be outlined in a future article).
• Nudging language is specific to a course and each activity to be nudged.
• Nudges are more effective when they follow a promotion of the key resource and are limited in number.
• Nudges can help to give a "teacher presence" to online courses, and as such, the wording of nudges is important.
Positive outcomes included increases in team members' professional learning and development. For example, a number of team members who had not been involved in the earlier nudge project developed their understandings and increased their appreciation for the potential ways to use and interpret CLAD data. The multidisciplinary team also gained collective insights into strategies for designing, implementing and refining communications, such as initial promotion and nudging of key resources to support student engagement, particularly with low and non-engaged students. From these learnings the team were able to develop a set of Nudge Guidelines to provide a step-by-step guide detailing how to design and implement a nudge communication strategy for increasing student online engagement, and to consider how important the style and tone (wording) of the nudges was to students' engagement in their courses. After the approach was refined for S2, it was applied in a much more uniform and targeted manner across the courses. At the end of S2, the guidelines were also further refined into a onepage "how to" document. By refining and shortening the guidelines, it was felt the 'how-to' guide would demonstrate that the implementation of a nudge strategy does not need to be an onerous, nor time-consuming task, thus making nudging more accessible to other USQ staff. The guide provides USQ teaching staff with a simple, step-by-step guide about how to implement a nudge strategy, breaking the process down into 4 simple steps that include determining what to nudge, planning when to nudge, identifying who to nudge, and lastly considering the style (or wording) of the nudges.

Conclusion
Much of the research specifically exploring and describing academic interventions that enhance learner engagement in the online space are relatively limited (Stone, 2019). To address these drawbacks, this article explored course specific online strategies and tests their outcomes and impact across a range of disciplines. That the findings are positive, is important especially given that they relate to the rapidly expanding and increasingly problematic online space for students (Stone, 2019). Another benefit is that, for academics, the nudge intervention is a proactive and relatively accessible and simple strategy for academics to employ and verify (especially with the Nudge Guidelines to assist them) as well as being easily contextualised or adapted across courses. This verifies the intervention's effectiveness in enhancing students' achievement in increasing numbers across disciplines, schools and faculties. In turn, students' potential for retention and success are amplified by their academics' capacities to assist them to more effectively access and employ critical resources for assessment requirements, align their expectations more realistically with those of the course and more effectively accomplish their learning obligations and responsibilities.
The project has also stimulated additional research questions. One relates to whether, with the contemporary push for engagement, students were experiencing "engagement fatigue" and whether this was becoming a phenomenon worthy of investigation. This was prompted initially by the recognition that a fine line exists between nudges and nags. In their reflection of the project one team member noted: Something to be careful of is sending students too many communications (promotions and nudges) as this has the potential to lead to 'engagement fatigue'. I think that our agreement as a group to only send out one nudge per resource and not overwhelming students with multiple nudges about that one resources was a major 'a-ha' moment in our learning as the group, as was the decision to promote and then nudge; and have a specific plan for how to carry that the promotion-nudge schedule.
Another question relates to the label of a "nudge" or nudging. This term is being assigned to institutional projects and the project team wondered if an alternative term needs to be considered for course-based interventions? For example, Dart and Spratt (2021) in their consideration of personalised emails in two first-year mathematics courses as a scalable strategy for supporting students with diverse needs, used the use the term "personalised emails".
A third question evolved from the question of what constitutes meaningful engagement. While it is acknowledged that accessing online course materials does not necessarily guarantee meaningful engagement, student access to learning materials is a factor that affects achievement (Romero et al., 2013;Soffer & Cohen, 2019). Further, student's decision-making in accessing a certain online course resource not only reflects the value they have placed on this resource, but arguably one type of evidence of their online engagement. As observational behaviours are a necessary but not a sufficient factor in engagement, they are indicative of the potential to be engaged (Dixon, 2015). Access to resources is also recognised in the online engagement scale, which reflects the main ways in which students engage online (Krause & Coates, 2008). Data regarding student access can be used to learn about student involvement (Soffer & Cohen, 2019) confirming that this is a topic which could also require investigation.