An analysis of user engagement in student Facebook groups

* Analysing the engagement of students in university-based Facebook groups can shed light on the nature of their learning experience and highlight leverage points to build on student success. While post-semester surveys and demographic participation data can highlight who was involved and how they subsequently felt about the experience, these techniques do not necessarily reflect real-time engagement. One way to gain insight into in-situ student experiences is by categorising the original posts and comments into predetermined frameworks of learning. This paper offers a systematic method of coding Facebook contributions within various engagement categories: motivation, discourse, cognition and emotive responses. *This ‘New Idea and Emerging Initiative’ was first presented at the 2015 STARS Conference in Melbourne, Australia in July 2015 and was selected by the Conference Committee as one of the top-rated reports. The authors have kindly given their permission to have this report published in the conference issue of the Journal and it has undergone a further review by the editors to confirm it aligns with the Journal format.


Introduction
Facebook communities provide students with an accessible and interactive forum for discussion on relevant social and disciplinerelated needs. Facebook groups can serve as a rich source of student data, chronicling their experience and relationships. The process of sifting through this data to make sense of the complex interactions can be a complicated task, so it is suggested that the utilisation of systematic analysis methods can help to streamline the practice. This paper puts forward one such methodology, a categorisation system, as a way of distilling user-engagement from the Facebook interactions within student peer communities.
In 2014, the Queensland University of Technology's (QUT's) Peer Programs undertook an analysis of one of the existing Facebook collaborative learning groups which had been established for students to share their knowledge and seek advice at a course-based scale. The analysis was conducted under the Peer Programs broad aim of identifying and supporting existing and emerging peer initiatives across QUT to facilitate and investigate leverage points for student success. In the course of performing this cohort-specific analysis, a step-by-step approach to the evaluation of Facebook groups was also developed to better understand the nature of student engagement based on the original student Facebook contributions.
The cohorts originally under examination were not, in this instance, asked any direct questions about their engagement, so all indicators of their experience were derived from subsequent analysis of their online conversations. As such, this research approach can be seen as a method of determining student experience from observation only, and could potentially be applied to other records of student interactions such as social media, blogs and transcripts of verbal conversations.

On line peer-learning engagement
Student engagement encompasses the time and effort dedicated towards successful learning outcomes (Kuh, 2009). Combined with the concept of peer-learning, this form of engagement involves students assisting each other in a shared experience. The benefits of students engaging positively in their learning processes is well documented (Rovai, 2002;Zepke, 2013); the central idea being that motivated students involved in their own development will more likely be successful students. Research has reinforced the validity of this theory with studies for school-age (Newmann, 1992), undergraduate (Kuh) and post graduate (Rockinson-Szapkiw, Heuvelman-Hutchinson, & Spaulding, 2014) students concurring. Descriptions of the ways in which peer-learning engagement can be categorised differ. Guthrie and Wigfield (in Kamil, Mosenthal, Pearson, & Barr, 2014) for example, states that engaged learners coordinate their strategies (cognitively) within a supportive community (socially) in order to fulfil their goals (motivationally). Similarly, Fredricks, Blumenfeld and Paris (2004) categorise the concept into three key ideas: behavioural, cognitive and emotional engagement. They argue that due to the multifaceted nature of the student experience, each of these concepts can potentially be present simultaneously. According to Fredricks et al., behavioural engagement is closely associated with the idea of participation, encompassing student involvement in academic and social activities. Similarly, cognitive engagement also involves the degree to which a student invests time and energy into their studies, but more specifically involves the mastery of difficult tasks and complex ideas. Lastly, emotional engagement encompasses the range of attitudes a student may have to their learning environment, peers and teachers. Another way of conceiving this approach is to consider that evidence of behavioural engagement can be found in what a student does; evidence of cognitive engagement can be found in how well a student does; and emotional engagement is evidenced by how much the student appreciates the experience.
The internet has provided new opportunities within which student engagement can occur, allowing students to connect with each other, with staff and autonomously with online learning material (Ekstrand, 2013). Coates (2007) highlights that this change in paradigm has altered the way in which students' connectedness and engagement occur in the university setting and suggests that ongoing analysis of such engagement is a necessary step in guiding future online and campus-based pedagogies. Rockinson-Szapkiw et al. (2014) found that the most popular online social media forum for undergraduate students was Facebook. In their study of United States college students, they estimated that between 85 and 90 percent of students were users of Facebook. They also found that, as a tool to engage and connect students with others and with their learning, Facebook demonstrated a strong catalysing influence. They state that Facebook users "demonstrated a stronger sense of connectedness with their peers than those who chose to interact with peers via the phone or email" (p.ii).

Development of a systematic analysis process
Determining student engagement within the online peer-learning context generally requires the deployment of qualitative and quantitative analysis techniques such as focus groups and user surveys. For instance Coates (2007) surveyed over one thousand Australian undergraduate students in his study of online student engagement. In this questionnaire, students were asked questions relating to the extent, meaningfulness and relevance of online tools in each student's experience. While valuable insights can be gained from such surveys, this type of approach tends to rely on participants' retrospective perceptions. However, the passing of time may potentially alter such perceptions of past events. For example, once a student has gained a passing grade, perhaps the anxiety they may have expressed in Facebook posts earlier in the semester has dissipated, leaving only memories of a satisfied completion. In this case, the original Facebook posts themselves will tell a story which more truly reflects the sentiments of the student community at the time the experience occurred.
For the purposes of analysing the engagement of students within specific student Facebook groups, an approach was developed which utilised the original posts and also managed to cope with large data-sets. The speed and brevity with which Facebook contributions are generally made and the potential for a large number of students to participate can lead to the accumulation of hundreds, if not thousands of posts and comments throughout a university semester. For example, QUT's Caboolture Education Facebook group attracted 2,922 contributions from 73 individuals over the course of a year. To analyse the Facebook group participation records, the raw data was first downloaded from the Facebook website using NVivo software, which facilitates the taking of publicly available data without programming expertise and arranges it into exportable spreadsheets.

Coding the data
Given the potential size of the Facebook data files, the use of automated NVivo procedures to highlight patterns in user experience would appear to have some merit. For instance, the NVivo Query tool allows users to find the location of particular terms, commonly linked words, synonyms and frequently occurring terms within a database. It also offers a variety of visualisation tools including word cluster analysis, tree maps, charts and word frequency arrangements.
While such automated procedures offer some insight into common topics of discussion within Facebook groups, indepth analysis is limited. For instance, while counting the occurrence of certain terms may indicate the presence of general themes of conversation, in isolation it is not definitive, nor does it highlight the complexity in each individual's reaction to such topics, nor the way that individuals feel about these topics. Likewise, relational diagrams of word distributions do not necessarily illustrate the full depth of participant experience.
Given the limited nature of automated qualitative data analysis, the manual coding of data is often also employed by researchers to extract meaning and explore abstract ideas. Coding can be described as the classification or indexing of certain parts of a text in order to facilitate comparison and retrieval (Bazeley, 2007). Any piece of text can potentially be labelled in a variety of ways according to the research focus. For instance, the sentence, "I enjoy chatting with friends on Facebook," could be categorised as a topic based on chatting, friends or Facebook. Alternatively, it could be coded by the tone of the sentence; a positive affirmation of the value of Facebook. The choice of coding categorisation is thus a crucial step in the quantitative analysis process because, as Strauss (1987) points out, "the excellence of the research rests in large part on the excellence of the coding" (p. 27).
Hahn (2008) suggests that qualitative researchers must find meaning in their rich and complex data-sets through intelligent organisation, "without the time-tested statistical tools that dominate the world of quantitative analysis" (p. 1). Richards (1998) suggests that contemporary software enables the researcher to establish both a closeness and distance, by better facilitating rapid data interrogation (closeness) and improving upon summarisation techniques of overall results (distance). An integrated approach is thus one of the options now available to the qualitative researcher, where quantitative methods can be applied to a qualitative project.

Selective coding descriptors
The approach of Fredricks et al. (2004) to evidencing student engagement can be described as what a student does (behavioural), how well a student performs (cognitive) and how they appreciate the experience (emotional). Applied to the engagement of students in Facebook groups, these three aspects can also be incorporated by way of analysing the content and motivation behind student Facebook contributions to inform behavioural characteristics; by comparing student grades for cognitive evidence and through an evaluation of the emotional tone of each contribution. While a comparison of student grades against the number of their Facebook posts and comments results in statistical analysis only, the other forms of evaluation require informed coding choices to translate raw data to qualitative interpretations.
Two coding descriptors were selected to help define the behavioural characteristics of Facebook contributors: motivation and discourse. Motivation refers to the underlying reasons which may have led to a student's decision to post or comment on the Facebook site. Discourse refers to the actual topic of conversation, once the decision to engage has been made. Consequently, behaviourally, the combination of these two elements encompasses the initiation of engagement and the course of discussion. Alternatively, emotional tone was deemed to best gauge emotive reactions. Bazeley (2007) describes this coding of ideas into identifiable units as categorisation. She states that while higher order groupings of ideas may be considered concepts, the term category is best suited to descriptive subsets of concepts. So in this case, at a conceptual level, the central idea revolves around student engagement, while at a categorical level, the key ideas can be best described as behavioural motivation, behavioural discourse, cognition and emotional tone.
A number of coding sub-categories were also identified within the context of each category. For instance, under motivation, the subcategories are question, answer, thank you, encouragement and unprompted. While the categories occur concurrently, the subcategories are mutually exclusive. For instance, there are concurrent behavioural, cognitive and emotional dimensions to each Facebook contribution. However, for clarity of comparison, the choice of sub-categories relies on mutually exclusivity, so each contribution can only be coded to one topic. Where potential overlaps occur, judgements thus need to be made in choosing one sub-category over another. For example, if a student comments, "Thanks for your help with clarifying the study notes, together we can all pass this exam," primarily this is a thank-you even though it seems to also offer elements of encouragement. The choice of these five sub-categories was thus made on the basis of best-possible mutual exclusivity, so that all possible contributions would be included but conflicting overlaps in coding would be reduced. The sub-category unprompted, tends to act as a catch-all in this category for any contributions which are not questions, answers, thank-yous or encouragement. The same approach was taken in the choice of sub-categories for behavioural discourse. This topic revolves around the idea of student participation and includes participation in academic and social activities (Fredricks et al., 2004). These include assessment item, course content, university processes and social interaction. Lastly, emotional tone includes positive, neutral and negative sub-categories.

Allocating codes
The process of coding involves the manual allocation of one sub-category to each Facebook post and comment. For example, if a student posts, "Does anyone know what day we submit our first assignment?" the appropriate subcategories would be: question (behavioural motivation), assessment item (behavioural discourse) and neutral (emotional tone). The coding of Facebook engagement subcategories can be achieved in NVivo software but this may not be the most effective platform. NVivo is well-suited to large tracts of text such as from qualitative interviews that must be analysed with multiple overlapping codes. However, the brevity of most Facebook contributions and the mutually exclusive nature of the coding subcategories suggested by this research, means that NVivo may be too cumbersome for the project at hand. Instead, spreadsheet software such as Microsoft Excel may actually offer simpler and more straight-forward functionality for the task, as well as more-easily facilitating diagnostic analysis and graphic representations such as histograms once the coding is complete.
Engagement and connectedness can influence students' decisions to persist in their chosen courses, so analysis of such practices can help tertiary institutions to better deliver material to their students and facilitate practices for clearer and more effective communication, ultimately helping to build student success. Peerorientated Facebook communities are one such platform for student engagement in their learning so the analysis of interactions as they occurred can offer indications of the nature of their involvement at a number of levels. This paper recommends that such analysis revolve around the three pillars initially developed by Fredricks et al. (2004): behavioural, cognitive and emotional engagement. This novel approach responds to a need to analyse the realtime data, rather than seeking a retrospective student voice through surveys.

Questions for consideration
 Do the proposed categories adequately cover the potential depth of student engagement?
 In the development of sub-categories, is mutually exclusivity important?
 Is there automated coding software available (apart from NVivo) which can adequately cope with the subtleties of language in differentiating coded entries?