Evaluating nursing students’ engagement in an online course using flipped virtual classrooms

Flipped classroom models allocate more time for active learning approaches compared with more traditional pedagogies, however what is less clear with the utilisation of flipped learning is evidence to support whether students in flipped classes are given more opportunities to develop higher order thinking skills (HOTs) to effect deep learning compared with the traditional ways of teaching. Focussing on this gap, this study compares on campus and off campus student engagement in two courses using different deliveries: online face-to-face (f2f) mixed mode (on campus students attend traditional f2f on campus classes and off campus students study exclusively online) versus fully online mode, utilising flipped classes (all student study off campus engaging in flipped virtual classes). Final course grades were similar for both deliveries; however, the study suggests flipped classes offered students more opportunities to develop HOTs and engage more deeply in the learning process. Students’ evaluations of the online flipped delivery were mixed, with those students previously enrolled exclusively as on campus, particularly dissatisfied with fully online delivery and virtual class tutor experience. Recommendations are made concerning both the timing of the introduction of fully online delivery in a program and the need for continual up-skilling of staff who teach in online environments. Please cite this article as: Phillips, C., & O’Flaherty, J. (2019). Evaluating nursing students’ engagement in an online course using flipped virtual classrooms. Student Success, 10(1), 59-71. doi: 10.5204/ssj.v10i1.1098 This article has been peer reviewed and accepted for publication in Student Success. Please see the Editorial Policies under the ‘About’ section of the Journal website for further information. Student Success: A journal exploring the experiences of students in tertiary education This work is licensed under a Creative Commons Attribution 4.0 International Licence. As an open access journal, articles are free to use with proper attribution. ISSN: 2205-0795 Evaluating nursing students’ engagement in an online course using flipped virtual classrooms 60 | Student Success, 10(1) March 2019 Introduction Currently the challenge in nurse education is to make programs convenient, accessible and attractive to a wider cohort of students. Elearning and blended learning have the potential to meet this challenge (Delialioglu &Yildirim, 2008; Limniou, Schermbrucker & Lyons, 2018). However, there is mixed evidence supporting whether fully online electronic delivery is suited to all students and indeed whether all students are satisfied with this form of delivery. Many students have preferred f2f learning as this method of teaching directly supports enhanced shared understandings, development of interpersonal relations and communication skills, qualities not easily replicable online (Paechter, Maier & Macher, 2010; Price, Richardson & Jelfs, 2007; Smyth, Houghton, Cooney & Casey, 2012). In addition, there is a lack of robust comparative research that investigates students' experiences of blended delivery, utilising a combination of f2f and online learning components, compared to fully online electronic delivery (Bliuc, Goodyear & Ellis, 2007; Paechter & Maier, 2010; Rotellar & Cain, 2016). This has been attributed to methodological problems caused by the vast number of blended learning approaches developed and the proportion of online and f2f components varying for every course (Guzer & Caner, 2014). The current study attempts to address some of these challenges by comparing student outcomes in two courses offering similar content but delivered using a traditional blended (online-f2f) learning approach to one offered fully online utilising flipped virtual classrooms.


Introduction
Currently the challenge in nurse education is to make programs convenient, accessible and attractive to a wider cohort of students. Elearning and blended learning have the potential to meet this challenge (Delialioglu &Yildirim, 2008;Limniou, Schermbrucker & Lyons, 2018). However, there is mixed evidence supporting whether fully online electronic delivery is suited to all students and indeed whether all students are satisfied with this form of delivery. Many students have preferred f2f learning as this method of teaching directly supports enhanced shared understandings, development of interpersonal relations and communication skills, qualities not easily replicable online (Paechter, Maier & Macher, 2010;Price, Richardson & Jelfs, 2007;Smyth, Houghton, Cooney & Casey, 2012). In addition, there is a lack of robust comparative research that investigates students' experiences of blended delivery, utilising a combination of f2f and online learning components, compared to fully online electronic delivery (Bliuc, Goodyear & Ellis, 2007;Rotellar & Cain, 2016). This has been attributed to methodological problems caused by the vast number of blended learning approaches developed and the proportion of online and f2f components varying for every course (Guzer & Caner, 2014). The current study attempts to address some of these challenges by comparing student outcomes in two courses offering similar content but delivered using a traditional blended (online-f2f) learning approach to one offered fully online utilising flipped virtual classrooms.

Virtual classrooms
A primary concern with online learning is that most students miss out on real-time collaborative learning opportunities availed to their f2f counterparts. A virtual classroom (hereafter abbreviated to VC) is an electronic platform in which students can collectively interface verbally and synchronously with their tutor thereby reducing the 'transactional distance' that has been reported in connection with learning in fully online courses (O'Flaherty & Laws, 2014;Stein, Wanstreet, Calvin, Overtoom & Wheaton, 2005).
Many studies report major benefits of using virtual classrooms such as providing immediate feedback, encouraging exchange of multiple perspectives, enhancing dynamic interactions among participants, strengthening social presence, fostering the exchange of emotional supports and supplying verbal elements (Kear, Chetwynd, Williams & Web, 2012;Park & Bonks, 2007). However, fewer studies have specifically evaluated nursing students' experiences and outcomes after engagement with e-learning in a VC (O'Flaherty & Laws, 2014). The current study aims to address this gap in the literature.

Flipped classrooms
The flipped class approach was first popularised in secondary education (Bergmann & Sams, 2009) and has a continued focus in higher education (Pluta, Richards & Mutnick., 2013). The flipped approach has many advantages for students; it allows learning to be independently paced, flexibility of when and where electronic resources are accessed, and actual class time is used more effectively to engage students in dynamic discussion and interactive learning. The flipped approach allows students to practice lower order thinking skills (LOTs) such as remembering and understanding independently, and at their own pace. Class time can then be guided by tutors and peers, to devote time to learning activities allowing students to develop higher order thinking skills (HOTs) such as application, analysis, synthesis and evaluation (Bergmann & Sams, 2009;Lee & Lai, 2017).
A number of studies have found that student evaluations were more positive with the flipped approach rather than the traditional way of teaching (Butt, 2014;Fulton, 2012), while others have found that students' satisfaction was higher for traditional ways of teaching (Love, Hodge, Grandgenett & Swift, 2014;Strayer, 2012). What is less clear with the utilisation of flipped learning is strong evidence to support whether students in flipped classes are given more opportunities to develop HOTs to effect deep learning compared with the traditional ways of teaching (Hung, 2015).

Scoping the literature
In a previous scoping review on the use of flipped classrooms in higher education, O'Flaherty and Phillips (2015) noted very few articles that used a robust scientific approach to evaluate educational outcomes in flipped classes as it related to improved student learning, particularly acquisition of higher order skills such as problem solving, inquiry and critical or creative thinking. The review also found conflicting results regarding its pedagogical acceptance by staff and students and little evidence to show whether the flipped approach is best introduced in first or third year courses.
As VCs offer an effective means of mimicking the positive qualities of f2f teaching (O'Flaherty & Laws, 2014), the authors reviewed the literature for contemporary evidence of nursing students' experiences and related scholastic outcomes using flipped teaching in VCs. Although Betihavas, Bridgman, Kornhaber and Cross, (2016) completed a systematic review on the use of flipped classrooms in nursing education and identified five studies that evaluated outcomes associated with this flipped style of teaching, none of these studies used virtual classrooms to deliver flipped content.
The research reported in this paper thereby adds to the literature by evaluating whether the use of a flipped teaching approach in an exclusively online course, facilitated through VC delivery, is applicable to a final third-year nursing capstone course and whether the approach is embraced by students and teaching staff to transform curriculum.

Aims
The context of this study reflects the move by the researcher's university to deliver a final capstone nursing course from a previous online-f2f mixed mode offering of 12-week duration, to one that is now facilitated exclusively online for eight weeks. In previous online -f2f mixed mode deliveries of the course (a mode of learning that combines f2f classes with access to online resources) students were typically enrolled as either on campus (internal) -receiving blended f2f education, e.g. on campus tutorials, VC recordings, and lectures with additional online resources, or as off campus (external) -receiving resources and tuition totally online, e.g. utilising online synchronous virtual classes, VC recordings and vodcasts (voice synchronised with power point presentation) of lectures. This mode of delivery will hereafter be referred to as online-f2f mixed mode.
The new course is offered fully online utilising flipped virtual classes as it was deemed an effective, convenient and accessible alternative to deliver the same course content over the shorter eight week's duration, with students on concurrent clinical placement (Critz & Wright, 2013;O'Flaherty & Phillips, 2015). This mode of delivery will hereafter be referred to as online flipped.
The current study aimed to examine differences between these two courses that use different delivery modes (online-f2f mixed mode versus online flipped) in terms of: (1) final course grades (2) student satisfaction and (3) learning attitudes.

Research design
This study used a mixed methods design to collect two data sets. One data set utilised the online software program, SurveyMonkey®, for survey design and collected responses from students engaging in the new online flipped delivery both pre-and post-course from August to November 2016, respectively. When answering the pre-survey, students were advised to consider a prerequisite course they had just completed which offered a more traditional non-flipped teaching lesson plan. The survey instrument consisted of an electronically based 25-item, five-point Likert Scale questionnaire, designed to assess student demographics, with five additional open-ended questions, designed to assess the students learning approaches and interactions with preclass resources, as well as delivery preferences.
The researchers were interested in whether using a flipped approach as a conceptual framework enhanced student acquisition of deeper learning attitudes (i.e. HOTs). As students in the new online course had not previously been exposed to flipped teaching, the authors used responses in the pre-course survey to gauge their usual learning attitudes and pre-class preparation. The post-course questionnaire was used to gauge any change in their learning approaches after being exposed to a flipped teaching approach used in the new online course. The modified Study Process Questionnaire (SPQ), first developed by Biggs, Kember and Leung (2001), and subsequently modified by Hung (2015) makes a distinction between deep and surface learners.
The second data set was collected using the authors' university course evaluation instrument (CEI) which is a standard five question response with both five-point Likert type scales questions and free text responses on how satisfied students were with the overall course. The CEI data was collected from one cohort of students post-course in November of 2016, when the course was offered in its new exclusively online format. This was compared to student cohort's responses post-course from November of 2015 when the course was offered in its previous online-f2f mixed mode.
The human research ethics committee at the governing university approved this study.

Participants
The two sets of participants in this study (N=635 in 2015 online-f2f mixed mode; N=650 in 2016 online flipped delivery) were recruited from two separate offerings of the capstone course in a Bachelor of Nursing degree at an Australian university, which focused on preparation for transition to registered nurse practice. Student demographics were similar for the two cohorts. The online offering reflected the following demographics: 91% female, 77% domestic, 23% international, 62% enrolled internally, 38% externally enrolled. The demographics for the blended offering: 86% female, 80% domestic, 20% international, 60% enrolled internally, 40% externally enrolled.

Traditional teaching lesson plan
In the 2016 prerequisite course, students were required to complete pre-class core readings and watch videos before attending either their on campus tutorial (18 classes scheduled) or off campus VC (eight classes scheduled), but they did not have specific activities to complete before class time. Student numbers in both classes were comparable between 25-30 students per class. An online discussion board was used by the course coordinator to answer questions posed by students. The 90 minutes f2f classes were run every two weeks for 12 weeks. They were primarily tutor driven and focused more on student understanding of content than actively engaging students in collaborative learning, with the tutor addressing any questions pertaining to course content and driving the discussion points taken from preclass core readings. These classes acted as a scaffold for students to create and submit two final summative pieces of work.

Flipped classroom teaching lesson plan
After completion of the prerequisite course, the same students commenced the 2016 online course utilising flipped classes. In this course students met every two weeks, for eight weeks, for one 90-minute VC. The students could selfselect which VC, of the 26 on offer, to attend depending which time best suited their needs. Therefore, all VC were a homogenous mix of internal and external students. Student numbers varied between 15-25 students per class. Prior to attending a VC, students were required to complete two to three hours of the pre-class (flipped) activities that included core readings, viewing video clips, interpreting texts, self-assessment quizzes, research to address short answer questions based on clinical scenarios, and generating their own list of questions based on these activities to discuss and share with the group online. These activities served as a vehicle to equip students with LOTs such as remembering, understanding and application prior to attending the live class (Bloom, 1984;Lee & Lai, 2017).
At the start of the live 90-minute VC a short 5-10 online quiz, delivered utilising the VC polling feature, gauged whether students had engaged with the pre-class content. The VC format included addressing any questions pertaining to course assessments, discussion points taken from pre-class flipped activities where students could respond individually or work together in breakout rooms. The focus of the VC breakout rooms was on application of knowledge and development of HOTs These sessions were student driven with students discussing, sharing and asking questions of each other pertaining to the worksheets they had completed pre-class, and then all students regrouped with the tutor to ask further questions, answer questions, or respond to comments posed by the tutor. The VC content related activities followed on from the pre-class flipped activities and acted as a scaffolding exercise for students to explore and develop concepts, with an end goal of creating and submitting two final summative pieces of work.

Academic Achievement
There was little difference in students' academic achievement between the two course deliveries (online-f2f mixed mode versus online flipped). In 2015 students average final score for the online-f2f mixed mode course was 63.0% (internals 60%; externals 66%) with a course pass rate of 96% (after supplementary exams offered). In 2016 students average final score for the online course was 62.0% (internals 58%; externals 66%), with a course pass rate of 97% (after supplementary exams offered).

Student Satisfaction with course
Student ratings for the course evaluation instrument (CEI) question: "Overall I was satisfied with the quality of this course" were also compared between the two course delivery modes. Student satisfaction ratings for the online-f2f mixed mode course were 62% for internals (survey response rate (r) = 41%) and 58% for externals (r = 35%) compared to only 32% for internals (r = 40%) and 51% (r = 34%) for externals for the online flipped delivery.

Student perceptions of flipped virtual classroom delivery
Prior to commencing the online flipped course students completed a pre-course survey to provide feedback regarding their usual mode of engaging in more traditional non-flipped classes.
Post-course, these same students were surveyed with the same questions relating to their mode of engagement with the flipped classes. These responses are recorded in Table  1 (non-flipped) and Table 2 (flipped). Additionally, in the post-course survey, students were encouraged to state any benefits, gained from their VC experience and any problems encountered. The authors analysed students' responses to the set questions and open-ended questions within the context of five

Benefits of flipped learning
Comparisons of student responses (strongly agree/agree) to questions in Table 1 and 2 show a 30% subjective increase in student engagement with pre-class activities in flipped classes compared to non-flipped classes (question 1) and a similar increase in student preparedness for class activities in flipped classes (question 2). This result is also reflected in a student's free text comment: "Doing the preclass activities really helped me be more prepared and confident so I could participate more actively in the VC." Student responses (strongly agree/agree) to question 3 (Table 1 and 2) show a 20% subjective increase in student's critical thinking skills after engaging with pre-class flipped activities compared to in a non-flipped class.
Students free text responses highlighted they agreed that a flipped learning approach offered them more flexibility to work at their own pace compared with a traditional non-flipped approach, as well as enabling them to engage with more course content: "I could interact with the pre-class activities when it suited me".
In the post-course survey (see Table 2, question 5) 85% of students strongly agreed/agreed that they would recommend the flipped approach to other students. Free text comments included: "really enjoyed the work on my own before class then brainstorming, bouncing ideas, critically analysing work in VC" … "such a better way to learn." Although no formal analysis was conducted on the results from pre-class quizzes, tutors noted a correct response rate to questions in excess of 75%, indicating that the majority of students were prepared for their virtual class.

Challenges of flipped learning
Responses to question four ( Table 2) show 74% of students strongly agreed/agreed with the statement that engaging with flipped activities significantly added to their workload. Students perceived they devoted more effort to their current course pre-class preparation compared with their previous nursing courses, where only 47% of students strongly agreed/agreed that engaging in pre-class activities added significantly to their workload (Table 1). Student free text comments included: "so much more work!" in the flipped class.

Benefits of virtual class delivery
Student's free text comments relating to perceived benefits of the VC technology and aspects of VC delivery reflected that VC's supported the student's mode of study, for example: "Virtual classrooms were very flexible with timing as I could balance my time whilst at work, on placement and other studies and still attend a class." Students who had engaged with the pre-class flipped activities appeared deeply engaged in VC sessions, as reflected in student comments: "Knowing I was going into a VC made me complete my readings and flipped activities prior. This enabled me to be an active participant in the VC and online forums." … "Although typing takes longer than speaking, I could take time to formulate a reply in the chat area of the VC and did not have to respond immediately unlike my on campus classes."

Challenges of virtual class delivery
One challenge of the VC centred on the awkwardness in communicating with others without nonverbal cues. This was also related to the time it took students to feel comfortable using the VC and as they attended more often they felt they could better participate. However, the major issue students reported with virtual class delivery was adjusting to different tutors teaching styles. Many students commented on a poor student-tutor VC experience. For example, some students commented on tutor unpreparedness and lack of tutor confidence in utilising the VC features to its full capability: "I find that the level of what I get out of the VC depends on how familiar the tutor is with the technology" … "The tutor totally took over the class."

Student course delivery preferences
One survey item questioned students preferred delivery of the third-year nursing course. Survey responses revealed that 63% of students would have preferred the course be offered as an online f2f mixed mode model, however it is worth noting that 90% of these responses were from internal students who would have only had previous experience with blended delivery in their program. It was difficult to pinpoint the particular reason(s) internals took issue with fully online virtual delivery. It could be an emotional reaction to having no control over how they enrolled in the new course; they may also find the fully online course more socially isolating than f2f mixed mode delivery. Internal students commented: "PLEASE go back to conventional on campus tutorials for internal students!" By contrast an external student commented: "I have enjoyed my time as an external and are very satisfied with fully online virtual delivery."

Attendance at virtual classes
Attendance at virtual classrooms was not compulsory, however 37% of students chose to participate in over 90% of the online classes. VC recordings were also made available to students who did not attend live classes, particularly due to clinical placement clashes. Over 70% of students regularly accessed the VC recordings each fortnight. Of those students unable to regularly attend a class, 82% of students strongly agreed that listening to the recording of a VC session was beneficial to their learning. Students who watched the recording could not view activity in the breakout rooms as this is not captured in Adobe Connect. However, a student spokesperson, promoted to presenter, would summarise their group's breakout room activity at strategic points during the VC for the benefit of the other participating groups which would also be duly captured on the recording for the benefit of absent students.

Student attitudes towards learning
Student's attitudes toward learning were measured before and after course completion using a modified questionnaire of Hung (2015). Student responses to three SPQ-related items in the questionnaires were exported to SPSS Version 23 and a two-sample Kolmogorov-Smirnov test for equality of distribution functions performed to compare pre-course perceived learning engagement in non-flipped classes to post-course engagement in a flipped class (See Table 3). This is a non-parametric comparison test which relies on no assumptions of normality nor homogeneity of variance. As student responses are self-reported the nature of the results are subjective in nature.
With respect to the three questions of interest that related to: deeper learning engagement (Q1), deeper learning strategies (Q2) and deeper learning motivation (Q3), the participants responses in the post survey scored higher in all areas in comparison to the pre-course participants (P < 0.0001).
These results suggest participants from the current course, using a flipped approach, were both more engaged with pre-class materials and in-class activities than in their prerequisite course, which adopted a less structured nonflipped teaching approach. Exposing students to the flipped teaching approach engaged the students in more deep motive and deep strategy learning approaches than those previously adopted in their traditional nursing classes.
The survey responses and free text comments suggest the flipped approach encouraged an increased sense of student self-efficacy. This is reflected in several responses to questions in the student questionnaires (see Tables 1 and 2), suggesting students study strategies and level of class engagement had changed post-flipped course compared to learning approaches used in their more traditional non-flipped courses. This positive student success in taking responsibility of their own learning by engaging in pre-class content was also noted by Koo et al. (2016) and similar developments of new independent learning strategies was also seen by McLean (2016) and Huber and Werner (2016).

Discussion
The current study attempted to answer three main questions. The first question dealt with whether a flipped virtual learning approach delivered exclusively online, improved students' academic performance over a more traditional online f2f mixed mode learning approach. The results show there was no significant difference in student academic performance between the two modes of delivery as reflected in similar student final course scores and pass/fail rates. There was little difference in academic performance on enrolment type (internal or external) between the two deliveries, a finding supported by others in the literature (O'Flaherty & Laws, 2014;Redpath, 2012).
The second question dealt with student's perceptions and overall satisfaction with the exclusive online delivery. Notably, students previously enrolled as internal, studying f2f on campus, had a particularly low acceptance of this mode of delivery; this was not reflected in external students' comments. Internal students have been used to on campus delivery of lectures and tutorials for the duration of their three-year program and as such blended delivery is both familiar and comfortable. The authors suggest that introducing exclusively online courses earlier in programs may mitigate these views, a finding supported by Green and Schlairet (2017) who used a flipped classroom approach implemented in the first course of an online undergraduate nursing program. Similarly, Mzoughi (2015) reported students perceived they were not getting value for money, unless they received f2f instruction from tutors. On the other hand, the authors  noted that the concomitant introduction of a flipped learning approach in this third-year online course was very well received by all students at such a late stage in their program.

Two sample Kolmogorov-Smirnov test for equality of distribution of students' perceived learning attitudes, to determine deep/surface learning attitudes, before (pre-flipped n=149) and after engaging in a flipped class (post flip n=155).
However, although many internal and external students reported positive benefits of both the flipped approach and the VC technology in the online course, they were not satisfied with their tutor experience. The tutors in this course were predominately casual, newly employed staff with minimal staff development and training opportunities in VC technology, particularly in how to use various features in the VC such as online polls, whiteboard, breakout rooms, and being able to switch effortlessly between layouts and presentations, a finding also reported by (O'Flaherty & Laws, 2014). The current authors also noted inconsistencies between tutors in terms of content they provided, leading students to go "tutor shopping" to find a competent tutor who was both content confident and consistent in content delivery, an outcome that has been reported a number of times with respect to both inquiry based and case based learning (O'Flaherty & Laws, 2014). Significantly, tutors require extensive experience in mentoring students in more fluid and spontaneous teaching and learning virtual environments, compared to the traditional f2f class, as this challenge itself can exaggerate any lack of content confidence in tutors. The study findings emphasise that a professional development framework is essential for tutors to develop competence and confidence in their online teaching to benefit students, and that this development is supported by the institution (Baran & Correia, 2014;Raffo Brinthaupt, Gardner, & Fisher, 2015).
As reported in the literature, for many students the workload in the flipped classroom seemingly increased (Ferreri & O'Connor, 2013;Strayer, 2012) although the current authors would argue only to the extent that was already expected of them by the staff. For some students, the flipped teaching approach may better align with how they prefer to learn, which is to obtain and study basic content on their own and use actual class time for interacting with the tutor, course content and their peers.
The third question examined whether engaging in flipped classes effected students' learning attitudes. Despite any significant improvements in students' academic grades after adopting the flipped approach, the results of the current study suggest that the group-based flipped classroom model adds a new element to help students learn more deeply, independently and from each other. This shift happens when students come to class, teaming up to work together on that day's assigned activities. This encourages student learning from one another, helping students to not only learn what the right answers are, but also how to explain to a peer why those answers are right, developing the students HOTs. Morueta, López, Gómez, and Harris, (2016) propose that social presence is important to build a community of inquiry for online students, leading to cognitive presence, where students construct meaning.
As students move toward their final courses in a program, they would be expected to adopt deeper learning approaches in achieving their goals. However, for many students online learning creates greater autonomy, which presents challenges to learners (Kovanovic, Gašević, Joksimović, Hatala & Adesope, 2015) requiring them to be self-regulated and motivated with learning (Broadbent, 2017). It is reassuring that in the current study the authors have observed, as per Hung (2017), subtle shifts in a more positive direction in students' learning approaches and cognitive skills after exposure to a flipped approach. Encouragingly, it is these skills that are important to instil in graduates the value of lifelong learning and other contemporary work place skills.

Limitations of current study
There are some limitations associated with this study. Although students seemingly changed their approach to learning over the duration of the online offering, this cannot be solely attributable to the changes in the flipped learning environment. For example, other factors that contributed towards the change in students' learning approaches (age, gender, prior academic achievement, preferred learning styles, individual tutor) and student challenges particularly with online group work, were not controlled for, nor independently assessed.
It may also have been beneficial to determine student self-efficacy pre-and post-the different delivery styles as separate groups -internal versus externals. The authors also acknowledge that the means by which the current study assessed improved deep learning is subjective and the 20% improvement is not an absolute result.

Recommendations
Due to the limitations, the findings in this study have been reported more in terms of recommendations than as absolute benefits or challenges of online delivery utilising a flipped learning approach.
Recommendations for best practice guidelines that have emerged are three-fold. Although the results provide evidence of a statistically significant change in students' approaches to learning more research is needed, perhaps using focus group interviews, to determine both how and what changes in the learning environment can have a practical effect on the way students' approach their learning tasks.
The second recommendation concerns the timing of the introduction of an offering of exclusive online delivery is. This study has highlighted that students, usually enrolled internally in nursing courses found it particularly difficult to adapt to fully online delivery and that the introduction of a new delivery approach so late in the program, regardless of the merits of its flipped design, is particularly difficult for internal students. As recommended by Dorrian and Wache (2009), both student and staff will benefit from improved communications in online courses such as providing clear, detailed instructions and creating appropriate student and staff expectations of the new model, and from scaffolding such learning approaches in earlier years of a program.
The third recommendation concerns the need for adequate student and tutor guidance and training for meaningful engagement in, and implementation of a flipped VC. The tutor's contribution to the flipped approach and their expectation and behaviour toward technology is paramount to the success of the flipped virtual model. Tutors should be provided with additional professional development opportunities and be more open to letting go of their traditional position of facilitator, to adopt more of a mentoring style. This would facilitate a greater student-centred learning approach, offering students more freedom to take active control of their learning.

Conclusion
The findings in this study suggest that flip teaching in a VC has the potential to more effectively engage learners in deep learning approaches compared to those in more traditional non-flipped virtual or f2f classrooms. However, although a flipped approach was viewed as successful by many students and may even have led to deeper learning strategies by some, the current study found no difference in student academic performance with the flipped approach.
The authors suggest that further investigation is required on the flipped classroom approach regarding tutors' characteristics and efficacy to students' learning process. An emphasis on a professional development framework for academic staff to effectively teach in exclusively online environments is essential, both for their own development and for transformative learning experiences for students.