When the Journey is as Important as the Destination: Time-Averaged Retention as an Alternate Measure of Student Engagement and Program Impact

University retention and completion rates underestimate true levels of student participation because of their reliance on measurements taken at commencement (or census date) and end of a program. As a result, these statistical snapshots miss what happens in between, failing to capture the true reach of the teaching and learning process, as well as the effort and resources involved. This is problematic when these numbers drive debate over higher education policy or institutional decisions over resource allocation. Here we propose a way of turning retention statistics into a more meaningful measurement of student participation, that we term engagement. In the context of this article, engagement is a calculated quantity based on the time-averaged student retention of a program or course. We argue that it addresses the shortcomings of snapshot metrics and provides some much-needed insight into student participation. We motivate its adoption and illustrate its use with worked examples, as a guide to practitioners, researchers and policymakers in the field.

. In these situations, where engaging with study is as much about personal discovery as it is about educational milestones, narrow definitions of 'success' or 'failure' in terms of completion prove inadequate (Bennett et al., 2012).
Thus, the problem is one of perspective (McInnis et al., 2000;Polesel et al., 2004). To the policy makers and funding bodies, higher education is a system. The simplest way to measure the efficiency of any system is to divide the output by the input (and interpret the shortfall as a loss). In this systems context, completion becomes a proxy for success (i.e. the success of the system in delivering education) which then becomes the benchmark for funding and policy (e.g. Department of Education and Training, 2017). But from the perspective of the individual, success is judged in the broader context of their individual circumstances.
The reasons why students leave undergraduate study are diverse and well-studied (e.g. Krause & Armitage, 2014 and references therein;Norton et al., 2018). They include institutional reasons such as teaching and student support, as well as individual student circumstances such as study load, academic preparedness and socio-demographic factors. It has long been recognised that institutions have a common role to play in strengthening individual persistence through personal adversity (Tinto, 1975). However, the dominant factors at play can differ substantially among educational providers and the communities they serve (e.g. Department of Education and Training, 2017). For the individual, the reasons for withdrawal are often manifold and compounding, and disentangling this interrelatedness to extract meaning is complex (e.g. Cao & Gabb, 2006;Ramsay et al., 1996). Cherastidtham, Norton and Mackey (2018) demonstrate how multiple factors combine into a cumulated risk profile that can change significantly as a consequence of individual education and life choices.
Several authors make a distinction between the positive attrition of students who voluntarily withdraw to pursue changed goals (e.g. employment placement, transfer to other education, self-discovery) and the negative attrition of students who never properly engage with their course objectives and drop out (Fralick 1993;McInnis et al., 2000;Norton et al., 2018). While it is desirable to minimise foreseeable attrition, non-completions due to unforeseen circumstances (both positive and negative) will always remain. An education system focussed on student betterment over retention (e.g. Tinto, 1993) will necessarily also find value in partial completion. Many students who do not complete still find their learning to be beneficial, despite not having obtained a formal qualification out of the time and cost incurred .
The value of partial completion is not reflected in current policy, presumably because of the way participation is counted through completion. To reframe policy discourse around the benefits of non-completion, institutions could start to report rates of partial completion as well as completion. The problem then becomes a mathematical one: how best to quantify participation of students who complete some fraction of a course (or program 1 ) of study without making it to the end. Ideally the statistic should be sufficiently general so as to be applicable to wide range of situations. It should also be straightforward to calculate and understand. Policymakers first need the ability to quantify both partial and full completions before they can begin to formulate policy accordingly.
The goal of this article is to demonstrate an alternative approach to the measuring and reporting of student participation, independent of completion. Our challenge is to create a new measure of fractional participation (like a completion rate) that takes into account the participation of those students who did not finish. We call our new measure the engagement metric. Our desire is that it be used alongside traditional measures of retention and completion when reporting on student participation.
Common benchmarks used to quantify rates of completion vary between policy makers, institutions and researchers (e.g. Department of Education and Training, 2017; Hodges et al., 2013;Norton et al., 2018;Wild & Ebbers, 2002). In the broadest terms, they all derive a rate by taking some initial count of students ('commencements' or 'enrolments'), subtracting a final count ('passes' or 'completions' or 'graduations'), divided by the same initial count. Sometimes re-enrolments are added to the final count. Common to all methods is their reliance on statistical snapshots at the start (or census date) and end of a program. As a consequence, they are unable to capture any information about what happens in between. For example, two programs with identical completion rates might have completely different patterns of retention. This is a problem when such statistics drive policy debate or resourcing.
For the purpose of this article we use the terms 'participation', 'retention' and 'engagement' in the following way. Participation is how many students are active 2 in a course or program at a given point in time. In this article, we do not concern ourselves here with how participation is measured, only that it can be in some satisfactory way. Retention is the fraction of students participating at a given moment in time compared to those at the start 3 . Engagement is also a fraction but one that considers the sum of all varying student participation as a fraction of the total possible. It is the main focus of this article. Table 1 summarises these definitions, illustrated with examples.

Table 1
The Definitions of 'Participation', 'Retention' and 'Engagement' Participation How many students are active in a course or program at a given moment. Activity can be measured in any way that is relevant to the particular circumstances. e.g. For a face-to-face course, activity could be attendance or submitted assessment or online presence, or some combination thereof.
For an entire program, activity could be measured by formal enrolment and final exam attendance data only.
The analysis in this article does not differentiate between courses or programs (because the mathematical reasoning is identical). Nor is it concerned with how activity is measured. Nor do we debate the suitability of one participation measure over another.

Retention
The fractional student participation at a given moment compared to initial numbers. (Exactly how we define initial is unimportant and does not change our analysis.) e.g. If there are 53 students participating in week 4 compared to 80 in week 1, then the week 4 retention is 53 80 = 66 %.
Retention taken at the end of a course or program is commonly called completion.

Engagement
The sum of all varying student participation as a fraction of the total possible. e.g. If there are 53 students participating in week 4 compared to 80 in week 1, (but we also know there were 70 in week 2 and 57 in week 3), then week 4 engagement is Motivating the mathematical and sociological desirability of using engagement as a measure of collective student success is the subject of this article.

Note. The definitions of 'Participation', 'Retention' and 'Engagement' as used in this article
The remainder of the article proceeds as follows. In the next section we define and discuss the mathematical motivation for the engagement metric. We illustrate the differences between retention and engagement measures for several different patterns of retention. We then generalise from the analytic form to the case of discrete time sampling. In Section 3 we illustrate calculations of engagement, by way of worked example. Section 4 discusses the use of the engagement metric to change reporting practices and reframe the policy discourse around the value of partial completion. A summary in Section 5 provides concluding comments.
Non-mathematical readers primarily interested in how to apply engagement may prefer to start at the Section 3 examples. They should also note Eqn. (8) in Section 5 summarising the calculation in words.

Mathematical Motivation: Time-Averaged Retention as a Metric of Student Engagement
Mathematically we define instantaneous student retention, R(t), as the fraction of students (0 ≤ R(t) ≤ 1) participating in a program or course at time t, (where 0 ≤ t ≤ 1 is measured as a fraction of the total expected time to complete). As noted above, discussion of exactly how participation is best measured is beyond the scope of this article. It then follows that at t = 0 our commencement enrolment is R(0) = 1 while at completion (t = 1) the fraction of students who complete is R(1). In the discussion below, we initially consider the theoretical case where R(t) is continuous and integrable, then discuss the equivalent discrete time sampled case.
Instantaneous student engagement at any time t =  is the product of the fractional student retention R( ) with the infinitesimal fractional duration of the engagement, d . Integrated over all course engagements up to some time t, the total fractional engagement, E(t), is While the latter provides a snapshot of student participation, the former accounts for participation over all prior time. As Eqn.
(1) counts the engagement of every student for as long as that student is engaged, it provides a truer reflection of the total collective student time invested in a course or program. We make no assumption about how R(t) changes over time, be it up or down. Figure 1 shows the differences between R(t) and E(t) graphically. Unlike the instantaneous R(t), E(t) embodies all participation prior to t by summing over area, divided by the area of the rectangle bounded by t.
The distinction between R(t) and E(t) becomes clearer by way of example. Figure 2 shows a comparison between the retention and engagement for six hypothetical courses or programs with different patterns of retention. The choice of curves reflects a range of plausible scenarios based on our higher education experience. In what follows we do not make a distinction between courses or programs: our mathematical methodology applies equally well to a 5-day summer course with 25 students as it does to a 4-year engineering degree program with 650 students.
All retention curves range from R(0) = 1.0 to R(1) = 0.5 and so represent programs with identical rates of completion (50%). However, the levels of student engagement between them are notably higher, with significant variation between cases. A program with a linear decline in retention (Fig. 2a) has a final engagement level of 74%, reflecting the time of engagement by all students, including those not present at the end. Retention with a steeper initial decline such as the exponential in Fig. 2(b) (half-life to final level, t½ = 0.2) still exhibits engagement levels exceeding retention by 10% throughout. Sudden mid-course changes in retention either up (Fig. 2d) or down (Fig. 2c) do not dramatically alter engagement, as we would expect for a timeintegrated quantity like the latter.
The biggest differences between retention and engagement are for those courses that lose most of their students late (Figs. 2d,e). In these cases the 50% completion rate does not capture the large numbers of students engaged with a majority of the course. Thus it is not an accurate reflection of the impact of the program on these students, nor on the teaching resources consumed by it. Figure 2(d) is a dramatic illustration of how large offsets between E(t) and R(t) (some 35% in this case) can highlight increasing late loss. This is valuable strategic information for institutions engaged in proactive retention initiatives.

Figure 1
The

Example Retention and Engagement Curves for Six Hypothetical Courses
Note. Comparison between retention R(t) (blue solid line) and engagement E(t) (red dashed line) for six hypothetical retention curves: (a) linear decline, (b) exponential decline, (c) exponential decline with a mid-course drop, (d) inverted exponential decline, (e) double exponential decline, and (f) exponential decline with a mid-course rise. All retention curves range from R(0) = 1.0 to R(1) = 0.5. End-ofcourse engagement values, E(1), range from 0.12 (in the case of b) to 0.35 higher (in case d) than the 0.5 completion rates.
We now turn to the case of the discrete time sampling of real-world retention data. The discrete form of Eqn. (1) for n fractional retention measurements R1, R2, …, Rn in sequential order (0 ≤ Rn ≤ 1, for all n), is where En is level of engagement at the time of the nth retention measurement. Equation (2) is just the average fractional retention. It is approximate because we have assumed uniform time sampling which may not be reflected in real data. Equivalently, in terms of ongoing student numbers N1, N2, …, Nn , where N0 is the number of enrolled students at the commencement of the program. Figure 3 shows the same retention curves as Fig. 2 but with discrete time sampling. We have deliberately omitted some time points to show the quality of the approximation with censored data. Using Eqn.
(2) we find that the end-of-course values for En approximate to within a few percent of the corresponding E(1) values in Fig. 2, except when the sampling misses a major feature of the retention curve, such as the mid-course rise in Fig. 2(f). Data limitations aside, Eqns. (2) and (3) are suitable simplified forms for calculating program engagement from real time-sampled data. Figure 2.

Discrete Time Sampling Cases for the Same Retention and Engagement Examples Shown in
Note. Comparison between retention Rn (blue solid circles) and engagement En (red open circles) for the same retention curves as Fig. 1 with discrete time sampling at semi-regular intervals. Estimates of En using Eqn.
(2) are to within a few percent of their analytic values in Fig. 2 except for case (f).

Example Calculations of Engagement
We now give an example of calculating the fractional engagement for a course or program. Our hope is that practitioners and researchers in the field will use the engagement metric as an adjunct to retention and completion reporting, thereby revealing a fuller portrayal of the student experience and the impact of programs of study.

Example 1: 12-Week Face-to-Face Course (120 students)
Suppose we have a fictional 12-week course with weekly attendance figures of 101, 92, 74, 86, 72, 64 and 52 students. We have satisfied ourselves that attendance is a good measure of instantaneous retention, even though not all weeks are known. The commencement enrolment for the course is 120 and the final exam attendance is 60. Using Eqn.
(3) with N0 = 120 and n = 8 for our full set of attendance figures, gives: notably higher than 0.50 completion measured at the final exam (60 out of 120 in attendance). What this says is that while half the students completed the course, the fractional engagement with the course content was nearly two-thirds. This is despite the week-to-week variability in attendance (both up and down) and the increase in attendance for the final exam.
If instead we wanted to compute the engagement level earlier in the course (say, at the time of the n = 3rd attendance measurement), then: which is again higher than the 0.61 retention figure based on the latest attendance (74 out of 120). Even though the retention jumps to 0.72 the following week, the engagement stays relatively stable at 0.74, reflecting the longer term behaviour captured by this metric. Note that in all cases, we have not used the timing information of our measurements, except to ensure that each new datum is included in sequence.

Example 2: Two-Semester Pre-University Enabling Program (3250 Students)
The calculation is the same whether it is course or program. Consider the case of a hypothetical two-semester pre-university program with 3250 commencing students. If 2150 complete first semester, 2650 start second semester and 1950 complete it, then the fractional engagement across the program is: This is the same as averaging the fractional retention over the three measurements and reflects the increased numbers of students returning at the start of second semester. In comparison, the completion rates for first and second semester are lower (0.66 and 0.60 respectively) as is their average (0.63).
Of course, with so few retention measures, we know nothing about how much of each semester is utilized by those who leave, (and without additional data, nor can we). Even with one more retention measure each semester (say mid-semester test completions of 2450 and 2100) the engagement changes little: because the decline across each semester is uniform.

Re-Framing the Discourse on Student Success by Quantifying and Reporting Partial Completion
As discussed in Section 1, traditional discourses around non-completion as wasted opportunity largely ignore widespread evidence to the contrary, from both teachers (e.g. Fralick 1993, Polesel et al., 2004 and the students themselves (e.g. Bennett et al. 2012). The traditional system of accounting is presently unable to measure 'success' in any way other than full completion, and so policy definitions, funding mechanisms and institutional behaviour follow suit. We argue that if the benefits of partial completion are to gain traction in policy discourse, then the first step is to use and report retention statistics to measure it. If something can first be counted, it then becomes an entity for inclusion in discussion about future policy. This is particularly the case when much of that discourse relates to resource allocation and so is necessarily quantitative.
The engagement metric proposed in this article was mathematically motivated and explored in Section 2. In it we have sought a metric that specifically accounts for prior participation but is sufficiently general to be widely applicable in a variety of educational settings. We have also sought a metric that is fractional and so measures an 'efficiency' so that it can be used in the same discussion and reporting scenarios as completion. Institutional action in the areas of student admission, resourcing teaching and retention strategies could also change as a consequence of having this information. Engagement adds a value to institutional reporting that can aid decision-making about how and where retention efforts are best targeted.
Widespread adoption demands ease of calculation, and in Section 3 we illustrated this by way of examples. The main challenge is not the calculation itself, but with the collection of retention data and we suspect the systems already in place at most institutions would readily facilitate this. The question then is whether the extra computational effort is worth it. If we value partial completion enough and the retention data are largely in hand, then we argue that this small extra effort is justified.
The fact that engagement rates are higher may prompt suspicion that we are attempting to make completions look better than they are. To do this, however, is to confuse the purpose of each. The engagement metric is not meant as a substitute for completion (which measures exactly thatthe fraction of students who complete). Engagement is measuring the fraction who participate and so will necessarily be higher than the number who finish. Completion rates are lower because they fail to account for all of the intervening activity. One way of countering such scepticism may be by giving the completion and engagement difference as an offset. For example, if the completion is 0.50 and engagement 0.63 (as in Example 1 of Section 3), then the completion could be reported as 0.50 (+0.13). In this way, confusion can be avoided but the information content preserved.
Another potential suspicion is that the new metric is susceptible to gaming. By gaming we mean the manipulation of processes around measurement so that the metric is no longer a true representation of what was intended. As the engagement metric depends on retention statistics, then any gaming is no different to what could be done under existing systems. In this context, engagement is not creating any opportunities that are not already there. Any metric is susceptible to gaming.
Ultimately policy is framed and administered by what is measured. If policy focus is overly narrow or simplistic, then the discourse around it needs to be driven by more informative metrics. Identifying what is worthwhile to be measured is the first step, as the rich literature on positive attrition attests. The second step is to define the best way to measure it, as we have sought to do here. The third step is to adopt and use it widely, so that it becomes part of conversations about reporting and institutional decision-making. The final step is to apply the knowledge gained through its use into driving future reform.

Summary
This article argues that if partial completion of higher education is a beneficial endeavour of itself, then a metric that captures participation is needed. We make the case that a new metric (which we call engagement, E) is a useful adjunct to the usual measures of retention and completion. We argue that engagement should be reported alongside retention or completion wherever the latter are employed. Using engagement captures student activity and program impact in a way that instantaneous measures (such as retention and completion) do not. In this article we have not discussed the ways in which participation should be meaningfully measured, as it is beyond our remit. We have simply assumed that it can be, in whatever manner suits the particular setting.
Analytically, engagement is the time-averaged retention over the duration of the course or program up to that point. A useful non-parametric approximation to E is: ≈ (Sum over all student participation figures to date) (Number of participation measurements) × (Commencing enrolment) ( ) which assumes semi-regular sampling that captures any important changes in participation. The sparser the sampling and/or the greater retention fluctuation, the less precise the approximation. As an averaged quantity, engagement is generally higher than retention and changes more slowly.
The engagement metric can be calculated in a wide variety of settings and situations, including individual courses over a few weeks or semester, or entire programs over multiple semesters. It is independent of institution and the mode of program delivery, be it face-to-face, online or some blend of these. It can be reported alongside completion rates on its own (e.g. completion = 0.50, engagement = 0.63) or as an offset in contexts where its derivation is not commonly understood (e.g. 0.50 (+ 0.13)). Its susceptibility to gaming is no greater than that of the retention data from which it is drawn.
'Success' in higher education has long been quantified exclusively in terms of completions because they are relatively easy to define, understand and measure. The downside of this apparent simplicity is that non-completion becomes equated with failure, despite much evidence to the contrary. Positive attrition by students who voluntarily end their higher education to pursue other equally worthwhile goals, or who learn something about themselves in the process, is widely prevalent.
By having a metric that captures what happens during a program, discussions based on partial completion can be more fully informed. Our hope is that practitioners, program administrators and institutions will adopt it as an adjunct to completion reporting. At an institutional level, it could serve an important role in driving decision-making around enrolment and retention strategies. More broadly, it could reframe higher education policy debate around student success and the ways it is enabled and supported by institutions and government.