“I’m Not a Number, I’m Someone to Them”: Supporting Commencing University Students’ Through Technology-Mediated Personalised Communication

As universities offer more flexible delivery options there are parallel concerns about increasing levels of engagement. Withdrawal and disconnection from study is most common during the first year of university as students experience social, cultural and financial adjustments and attempt to understand the nuances of academic learning. This article reports on the impact of a technology-enabled support system delivering timely, personalised and actionable feedback on online activity and support emails at critical periods in two courses. Learning analytics data was used to identify appropriate engagement metrics for personalising feedback to students with results indicating an improvement in course grades. While the learning analytics approach provides a technology-mediated means for scaling personalised feedback and communicating with large cohorts of learners, the qualitative results indicated that students felt they were noticed as individual learners, were more willing to contact educators for support, and more motivated to engage with the online course learning materials.


Introduction
Engagement of commencing university students has been of ongoing concern, particularly with the increase in numbers of nontraditional students participating in higher education (Shah et al., 2016). Feedback is an effective way to engage students and improve their learning experiences (Pardo et al., 2017b), however flexible delivery options such as blended and online learning have required educators to find new ways to provide feedback and encourage online activity. This study describes the implementation of a learning analytics-based feedback system to support students' online activity in two first year courses. In addition to learner data, targeted communication tactics were implemented based on issues identified in the literature as affecting student engagement, such as academic challenges; negotiating work and study commitments; transition shock from school to university; and mental health and well-being (e.g. Baik et al., 2015;Brooker et al., 2017). Rather than targeting low/non-engaging or "at-risk" students (Lawrence et al., 2019), our intervention sought to increase the online activity and learning outcomes of all students enrolled across the courses. Drawing on students' engagement and performance data (i.e. interactions with the online learning platform, grades) and focus group interviews, we investigated whether course-specific As universities offer more flexible delivery options there are parallel concerns about increasing levels of engagement. Withdrawal and disconnection from study is most common during the first year of university as students experience social, cultural and financial adjustments and attempt to understand the nuances of academic learning. This article reports on the impact of a technology-enabled support system delivering timely, personalised and actionable feedback on online activity and support emails at critical periods in two courses. Learning analytics data was used to identify appropriate engagement metrics for personalising feedback to students with results indicating an improvement in course grades. While the learning analytics approach provides a technology-mediated means for scaling personalised feedback and communicating with large cohorts of learners, the qualitative results indicated that students felt they were noticed as individual learners, were more willing to contact educators for support, and more motivated to engage with the online course learning materials.
learning analytics feedback could be used to increase students' out-of-class online activity (as one indicator of their levels of engagement).

Review of Literature: Harnessing Technology to Support Student Success
Research has noted that withdrawal and disconnection from study is most common within the first year of university as students adapt to new social, cultural and learning environments (Araújo et al., 2014;Krause et al., 2005). As noted by Tinto (2009), support and feedback are two attributes of effective classrooms that promote success for commencing students. However, contemporary higher education is characterised by large enrolments and diverse cohorts, both of which present challenges for educators to provide personalised support and timely feedback. First year courses in particular, being introductory level, tend to draw large numbers of students with a diversity of disciplinary backgrounds and prior knowledge. In view of these twin challenges, learning analytics (LA) has emerged as a viable approach to scale the communication of personalised feedback and support (Pardo et al., 2017a). This review of the literature begins by tracing the key debates in student engagement in higher education, before looking specifically at the use of LA to support student engagement and success.

Student Engagement in Higher Education
Student engagement is accentuated because of its association with academic achievement (Kahu, 2013). In higher education especially, universities are increasingly under pressure to show greater accountability in terms of student outcomes (Zepke & Leach, 2010). Broadly speaking, the construct of student engagement is defined as "the time and effort students devote to activities that are empirically linked to desired outcomes of college and what institutions do to induce students to participate in these activities" (Kuh, 2009, p.683); these activities can take the form of interaction with faculty, interaction with peers, as well as wider participation in co-curricular activities.
The first year of university is a critical period for establishing student engagement, as it represents a significant shift toward a culture of independent learning. Students in their first year need robust support to help them acclimatise to the new academic environment, adapt to a culture of independent learning and academic rigour, and meet strict academic requirements in order to progress to higher levels of their degree programmes (Tinto, 2009). Moreover, contemporary higher education is seeing an increase in the adoption of blended and online modes of learning (Tai et al., 2019). Without support, or even the perception of such support, it can be challenging for students who are new to the higher education environment to know how to engage effectively for success (Fredricks et al., 2004). The use of LA may provide one way of further supporting students' engagement in university at scale.

Learning Analytics to Support Student Learning at Scale
Learning analytics is defined as the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs (Siemens & Long, 2011). The growth in online learning and the use of learning management systems in higher education institutions has resulted in the availability of a wide range of learner-related data (Baker & Siemens, 2014). For instance, when students navigate a learning management system (LMS), the system automatically keeps track of where users have been, the activities they have explored, or the resources they have accessed. The field of LA has emerged as a way to capitalise on the availability of large datasets, to bring a richer understanding of student learning and student engagement (Clow, 2013). A nascent development in education, reviews have already featured the promises of LA interventions for improving student learning (e.g., Sclater et al., 2016;Sønderlund et al., 2019).
Earlier applications of LA focused on predictive methods to develop early warning systems for identifying at-risk students (e.g., Arnold & Pistilli, 2012;Krumm et al., 2014). There has been increased interest to develop LA applications that close the loop for students, by providing students directly with their own data in the form of evidence-based feedback or nudges to support individual students' learning at scale. Laurillard's (2013) conversational framework supports this method of feedback. Laurillard's framework proposes that learning takes place through a series of conversations between the educator and students. LA-based feedback can support conversations, however, the framework highlights how reflection and adaptation are crucial to the communication cycle and assist with the creation of an effective learning environment. The communication cycle commences when the students interact with course learning activities and this interaction is captured by the LMS. Educators can then reflect on and make sense of this information which can be used to support communication in the form of ongoing feedback to students. Communication is fostered by students reflecting on the feedback they receive and actioning the feedback. These actions can promote future personalised feedback and support and new communication loops can be initiated through the adaptation of learning tasks (Pardo et al., 2018).
Presently, few studies exist to show the effectiveness of these newer forms of LA-based feedback on students' learning. This study aims to address this research gap by exploring the impact of personalised, LA-based feedback using the software tool OnTask (Pardo et al., 2018) on student online activity in two different courses. Three research questions framed our study: RQ1: What was the impact of the personalised feedback on students' online activity? RQ 2: What was the impact of the personalised feedback on students' course performance? RQ 3: What did students find helpful about the personalised feedback emails?

Participants and Courses
Participants in this study were first year students enrolled in 2019 in one-unit courses in an Australian university bachelor degree in either psychology (n=415) or communication and media (n=193). Students in the psychology course completed one online lecture (2 hours) and one practical (1 hour) weekly. Four practicals were delivered in class with the remainder online. Assessment consisted of eight online multiple-choice quizzes (10% weighting), a written article review assignment (50% weighting) and a final exam (40% weighting). Students in the communication course completed seven on-campus lectures (1 hour each), five online lectures (1 hour each) and one tutorial (2 hours) weekly. Eleven of the tutorials were delivered in class while one tutorial was delivered online. Students completed a continuous assessment consisting of two short answer reports (30% weighting), an essay (30% weighting) and a final project (40% weighting). Students in the communication course were enrolled in either internal or external classes enabling analysis of both cohorts. Given the student demographics were aligned from 2018 to 2019, and the assessments in both courses were the same, a comparison of the 2019 grades was measured against 2018 student course performance. At the start of the study period, students were advised the communications they were receiving were part of a research project that aimed to support first year students through technology-mediated personalised communications. It was explained that these regular communications were semi-automated and based on data about their individual engagement in the course.

Using OnTask for Personalised, LA-Based Feedback
OnTask (Pardo, et al 2018) serves as a repository for various sources of information about students' engagement, including activity data from the LMS, records of lesson attendance and course performance. Instructors can select specific online activity data for which to develop "if-then" rules to generate personalised feedback messages to all students. The system also serves as a vehicle to deliver these messages as emails. In both courses OnTask was utilised to provide students with personalised feedback regarding their activity and their performance in the assessments. For example, students received feedback regarding progress on quiz completions as well as nudges to access useful documents located on the course site such as assignment information, if they had not already done so. Students also received feedback regarding their grades in the assessments and, depending on their performance, they were either offered advice on how to achieve a higher grade and/or encouragement that they were doing well.
In addition to progress-related feedback, students in both courses were sent personalised support emails via OnTask at three critical points throughout the study period. In week 4 students were sent an email regarding work/study balance advising them of options around studying part-time. In week 5 students were sent an email with a focus on transition shock with study advice strategies. In week 7 students were sent an email regarding well-being and to advise them of the opportunity to attend a dropin session with one of the university counsellors. Feedback on the wording of these emails was provided by a small group of student academic representatives to ensure the messages were well positioned to appeal to the wider cohort. By way of example, the introduction for the week 4 email (providing part time and online only study options) is below:

Personalised Feedback and Support Points
It can be tough balancing your job with the demands of university study but you are not alone. Chances are the tutor you're currently working on that assignment for right now also had a job while studying … and 3 in 5 of your class-mates are also working while at uni. So it's good to know you've got options if the deadlines start piling up and you get that sinking feeling.

Learning Management System (LMS) Activity Data
To examine the impact of the feedback intervention on students' online activity (RQ1), LMS activity data for the courses was downloaded at the end of the semester. Feature selection techniques were applied on the raw data to extract meaningful attributes regarding online activity. In all, nine attributes were used (see Table 1). These attributes related to students' access to three resource types in the LMS: 1) Assessment refers to assessment-related content such as assessment information, exemplars, or templates, as well as assessment attempts; 2) Informing Orienteering refers to content regarding course-related information such as course information, announcements, timetable; 3) Learning Content Access includes weekly topic-related content, such as lecture recordings, readings, and formative, non-graded activities. Attributes 4 to 9 relate to students' use of time during their study, as time-use is an indicator of self-regulated learning and achievement (Hensley et al., 2018).

Course Performance Data
To examine the impact of the intervention on course performance (RQ2), data from the graded assessments was collected for both courses and compared against 2018 grade data.

Qualitative Focus Groups
RQ3 aimed to examine students' experience of the technology-enabled communication and what they found helpful (or otherwise) about the personalised feedback and support. Students from both courses were invited to attend a one-hour focus group to answer further questions regarding their experience of the feedback emails. In total, three focus groups were run (FG1=7 students; FG2=8 students; FG3=6 students). All students enrolled in both courses were invited to participate and asked to leave their email address with their course coordinator if they were willing to be contacted by the focus group facilitator to arrange attendance at one of three scheduled focus groups. All students who indicated their interest were invited to attend (25 students elected to be contacted and 21, in total, attended a focus group). While students from both courses were invited to participate, it is notable that only two students from the communication course indicated their willingness to be interviewed, with the remainder from the psychology course. Furthermore, the communication students did not attend the scheduled focus group and we acknowledge this as a limitation of the qualitative data as it only reflects the views of the psychology cohort. While we do not know the reason for this disparity, it may be that the psychology students had a greater interest in participating in research as research methods is one of the course topics and critical evaluation of research is the focus of the major assignment.
Focus group participants were asked to reflect upon their experiences participating in the course; the strategies they used to keep up to date and improve on the tasks required for the course; what role the teacher's input and feedback online/via email had on their learning and motivation; whether the feedback was helpful or not and in what ways; and what kinds of feedback they would have liked more or less of in the course. The focus groups were transcribed and analysed using thematic analysis techniques (Muir-Cochrane & Fereday, 2006;Stirling, 2001) in order to ascertain patterns in their responses. Thematic analysis was deemed appropriate because it is primarily concerned with the subjective experiences of research participants.

RQ1: What was the impact of the personalised feedback on students' online activity?
The psychology course was not included in this analysis as this course moved to a blended format in 2019 and therefore there were more online learning activities. As a result, student activity in 2019 was not directly comparable to the internal offering in 2018. To examine the online activity in the communication course (RQ1) data collected on these attributes were aggregated over the full study period. A session was comprised of successive learning actions that occurred within 30 minutes of each other. The following table illustrates how LA data was applied to the attributes. The median frequencies, minimum, and maximum values for the cohorts are presented in Table 2. From the result for AVG_Session, compared with the 2018 cohort, the 2019 intervention cohort had shorter sessions (Mdn = 4,531 sec, or 1.26 hrs) with the gap between login sessions (Session regularity) being smaller (Mdn = 11,419 sec, or 3.17 hrs) The intervention cohort also engaged less frequently with Assessment and Orienteering (Mdn = 36). The r-values indicate that these were all small-to-medium effects. Overall, these results indicate that the intervention cohort had more focused and regular sessions online.

RQ 2. What was the impact of the personalised feedback on students' course performance? Communication Course
Given the separation of internal and external classes this analysis was based on the 193 students who took the course in both modes. The proportion of internal students to external students was 83% (χ2 (1) = .482, p = .49, phi = -.04). Independent samples t-tests found that the cohorts performed significantly differently on two out of the five assessment components but were not significantly different in terms of their final course marks (see Figure 2). The cohort achieved a good result (M = 70.31, SD = 9.46) for the Short Answer 2 assessment, t(326.25) = 3.34, p = .001, eta-squared = .03 which was a small effect. The students also scored well (M = 70.09, SD = 16.5) on the Essay assessment, t(366.98) = 3.46, p = .001, eta-squared = .03, also a small effect. The datafile was split into mode of student (internal vs external), in order to ascertain if the cohort differences were influenced by mode of student. Independent samples t-tests found that performance between the two cohorts in internal mode, were significantly different for all the assessment components, except for the final course mark. Internal students from the intervention cohort outperformed their previous cohort peers in the Short Answer 2, Essay, and Project components.

Assessment Outcomes Communication Course
Chi-square test of independence found a significant association between cohort of student and course grade letter distribution, (χ 2 (6) = 17.54, p = .007, Cramer's V = .21) with an increase in the proportion of C grades (by 7.3%) and D grades (by 7.8%). Moreover, a decrease in the proportion of students failing the course was observed, from 15.0% to 11.4%. Interestingly, when the split-file analysis was conducted based on mode of study, only the association for internal students was found to be significant, (χ 2 (6) = 18.13, p = .006, Cramer's V = .23). The association for external students was not significant, (χ 2 (4) = 3.92, p = .42). This suggests that the effect of the feedback on academic performance was greater for internal students than external, however overall across most assessments the 2019 cohort performed better, and their overall course grades were stronger.

Psychology Course
This analysis was based on an enrolment of 399 students taking the course in a blended learning mode. Passing rates for the cohort was 77% (χ 2 (1) = .251, p = .62, phi = -.02. Chi-squared analysis for independence also found no significant association between cohort and distribution of grade bands, (χ 2 (6) = 2.97, p = .81, Cramer's V = .06). Figure 3 shows the student performance in the course assessments. The Psychology course had undergone some changes in 2019 with an increase in online teaching. This does not appear to have impacted on overall course performance. Students were not enrolled in distinct internal or external classes so analysis was of the whole cohort.

Assessment Outcomes Psychology Course
Together, the results indicate that the intervention cohort performed better, particularly in the case of the communication course. Furthermore, the results suggest that the intervention was more beneficial for internal than external students, in terms of course performance.

RQ 3: What did students find helpful about the personalised feedback and support? Qualitative Results
Focus groups were undertaken in order to understand how students experienced the technology enabled communications and what students found helpful (or otherwise) about the personalised feedback and support they received. There were two key forms of communication they received 1) individualised feedback reminding them to submit as assessment or letting them know about their progress in terms of online engagement and, 2) broader support communications providing information on university support services (e.g. learning advisers, study help, mental health and well-being services). All focus group participants agreed that the personalised feedback was the most useful, though around half of them did comment that the broader support emails were a great reminder of university services. Approximately half the focus group participants indicated that they felt they were often 'bombarded' during orientation week with information about university support services but then this communication tapered off, perhaps at a time when they needed it most (e.g. when their assignment load was heavy and for many, they were in the midst of juggling both paid employment and study). The need for general support emails was articulated through student feedback, for example: I guess because you get bombarded with all the new information right at the start, you forget certain things, so it's good then to have them actually reminded throughout the semester. (Focus Group Participant) Another key finding from the focus groups was that students felt that the personalised communications, in particular, motivated them to stay engaged and up to date with their learning and assessment. The communications also strengthened their relationships with their lecturer so that they felt more inclined to contact them if they needed support (this was raised by approximately seventy per cent of participants in each of the focus groups). The students felt that there was something unique about the consistent and personalised communication from teaching staff in their course and noticed that this was not something they were experiencing in their other courses. Students described a wide range of personal approaches to time management from checking their student portal daily for reminders, to planning out due dates at the start of a study period, to starting an assessment just before it was due. However, despite the spectrum of approaches to their course planning all students felt the regular feedback from the course coordinator supported them to keep on track. A key finding from the focus groups was that students felt that the communication helped them transition from the style of learning expected at school to that of university. For example: The thing is, for people who come here straight from high school, we're used to knowing everything because of our regular communication with the teachers. So it feels like we're coming into uni unprepared, where it's usually us having to figure it out online, last minute. So I think [the personalised feedback] was really nice for people who are coming from high school straight to uni, because they're used to. [regular communication] The pilot intervention appears to have developed students' ability to adapt to a new academic environment, which is crucial to their long-term success (Nelson et al., 2014;Tinto, 2009). This result also supports Sclater et al.'s (2016) argument that providing students with regular communication on their progress, and recommended actions to meet educational goals, fosters greater agency in their learning. Notably, approximately half of the participants in each focus group observed a flow-on effect, in that they began applying their adaptations to learning (around time management and engaging with the course sites and learning materials) to other courses.
Participants expressed that the personalised feedback increased their motivation to engage with the course and nudged them to complete their learning tasks (Graham et al. 2017;Lawrence et al. 2019). They did not feel the communications were overly intrusive or "clogged up their inbox". Some even reported that the personalised reminders "saved [their] grade a few times" because they submitted assessment that they otherwise may have missed. They also enjoyed the feeling that there was someone "looking over [their] shoulder" and that it "gave [them] and extra boost to complete everything" indicating not only was it important that the communication was regular but that the feedback was valuable because it also related to their own activity.
One of the most prevalent responses across the three focus groups (where there was agreement from all participants) was that, while the communications were mediated by technology, students still perceived the messages as personalised to them. An outcome of the perception of personalisation was an increase in likelihood of students to reach out if they needed help, as the lecturer felt "more approachable because [they] had that regular communication." However, an important point here is that the communication must be dialogic. While students were receiving supportive and personalised communication from the lecturer, the success of this strategy also relied on the teaching staff responding promptly and supportively to students if they asked a follow up question or responded to the communication they were sent.
In summary, the qualitative results indicate that students appreciated receiving support at two levels 1) Regular personalised feedback about individualised progress and, 2) broader information about university support services, with the former having a more notable effect on students' online activity. Secondly, students' motivation to stay up to date with their study was increased through the personalised feedback that nudged them to complete their learning tasks in courses both within and beyond this intervention. Thirdly, the communications opened a channel for students to reach out to educators and increased their confidence in communication with teaching staff. Finally, the communications eased the transition for students.

Discussion
This study aimed to examine the impact of technology-mediated personalised feedback and support on students' online activity and performance through a pilot intervention in two courses. Our results found that benefits for students in the communication course were clearer than those in the psychology course, as observed by higher achievement on some of the assessment components compared to a previous cohort without the intervention. A significant decrease was observed in the proportion of students who failed the course post census, as well as a significant increase in the proportion who achieved a credit or distinction grade post census.
Overall, it also appeared that students in the communication course logged in with greater regularity and may have been more focused in their online activities. This was in line with the feedback messages communicated, which encouraged students to ensure regular visits to the course site. One interpretation that could be made around this finding is that the regular and personal email communication about assessment, course due dates and sequencing of submissions meant that when they did log in to the course site, they used that time instead to engage with learning content (e.g., readings and lecture materials). Having more focus on the learning materials rather than managing assessment concerns could be seen as a benefit of this communication. These findings of different online activity patterns and increased academic performance provide further evidence of the impact of personalised feedback previously demonstrated by other researchers in this field (e.g., Lim et al., 2019;Pardo et al. 2017b). This finding was supported by focus group data which indicated that most students felt more motivated to succeed and engage with the learning materials. One unexpected finding for the communication course was that face-to-face students benefitted more than external (online) students (in terms of overall course grades). This finding suggests a blended learning model where external support was combined with face-to-face support, was of assistance to this cohort. While a positive outcome for blended approaches to student learning, future research is needed to consider how to support external students better, who lack the opportunity to engage with their lecturers face to face.
We acknowledge that this study is not without its limitations. The psychology course was changed to a blended teaching model in 2019 and there were an increased number of online practical classes which impacted on the activity data analysis. Given this change in pedagogical design, it was positive that the students received additional personalised messages and the course grades remained consistent rather than falling. As such, we strived to obtain data from multiple sources to investigate impact, and to address our research questions. We expect future studies will be able to build on this study to extend our findings.
In conclusion, this study contributes to the empirical base of research studying the impact of LA-based feedback interventions on student online activity and performance. We suggest that future research should trial this approach across whole disciplines focusing on first year commencing core courses. A useful next step would also be to examine the possible covariates of students' characteristics that will differentiate the impact (e.g. by program scores or grade point averages). Our study has shown that personalised communication and feedback informed by LA could be beneficial in terms of increasing the online engagement of commencing university students.