About the Author(s)

Betty Breed Email symbol
Department of Computer Science Education, North-West University, South Africa

Roxanne Bailey symbol
Department of Computer Science Education, North-West University, South Africa


Breed, B. & Bailey, R., 2018, ‘The influence of a metacognitive approach to cooperative pair problem-solving on self-direction in learning’, The Journal for Transdisciplinary Research in Southern Africa 14(1), a516. https://doi.org/10.4102/td.v14i1.516

Original Research

The influence of a metacognitive approach to cooperative pair problem-solving on self-direction in learning

Betty Breed, Roxanne Bailey

Received: 03 Nov. 2017; Accepted: 05 Apr. 2018; Published: 11 June 2018

Copyright: © 2018. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


Preparing students for self-directed learning has become an obligation of 21st century educational institutions. This study focused on developing students’ metacognitive awareness during cooperative pair problem-solving, with the aim of enhancing their levels of self-directed learning. The research involved a class of 33 second-year computer applications technology education students in a semester module. Quantitative data related to the students’ levels of self-directed learning and metacognitive awareness were gathered in a pre-test using Williamson’s self-rating scale of self-directed learning and an adapted version of Schraw and Dennison’s metacognitive awareness inventory, respectively. During the semester, the students implemented a set of metacognitive self-questions when doing cooperative pair problem-solving tasks. At the end of the semester, the same two questionnaires were used in a post-test. The results showed that students’ metacognitive knowledge and metacognitive control had increased, with both statistical and practical significance, especially with respect to procedural knowledge, planning and monitoring. The results further showed that the self-directed learning of students who initially scored moderate or low levels of self-directed learning had increased, with both statistical and practical significance in most areas of self-directed learning. Seven randomly-selected participants were then individually interviewed to collect qualitative data about their experiences and views of using the metacognitive self-questions during problem-solving in cooperative pairs, and how this influenced their self-directedness in learning. The results for the qualitative data correspond with what was found with the quantitative data. Based on these findings, we conclude that the use of the metacognitive self-questions during cooperative pair problem-solving positively influenced the students’ levels of self-directed learning.


Contemporary educational institutions need to equip students with appropriate competencies and skills that will enable them to succeed in a world characterised by rapid development and changes in technology, practices and careers (Francom 2010; Guglielmino 2013; Rashid & Asghar 2016). Traditionally, formal education could provide students with the knowledge and skills needed to do their jobs and practise their careers for a long period of time (Oswald 2003). However, the modern technologically complex information age necessitates an ability to continuously update one’s own knowledge and skills in order to keep up with development. This requirement of lifelong learning makes it essential that ‘people must be able to adapt to current societal changes and be able to self-direct the process of developing their own knowledge and skills’ (Oswald 2003:6).

Problem statement

For the contemporary student, mere acquisition of specific content knowledge while enrolled at an educational institution is not enough (Francom 2010). Educators have the obligation to support students to develop skills that will enable them to effectively adapt to changes, stay up to date and develop into self-directed, lifelong learners (Guglielmino 2013). The growing need to equip students with the skills to engage in self-directed learning (SDL) necessitates that educators explicitly attend to fostering these skills. Educators can foster SDL through instruction that provides students with opportunities to develop and improve their ability to direct their own learning (Oswald 2003; Song & Bank 2016). Dynan, Cate and Rhee (2008) adjudge SDL as necessary to develop students into lifelong learners.

According to Guglielmino (2013), educators’ contribution to promoting SDL includes implementation of teaching and learning strategies that are conducive to development of the skills and attitudes required for moving towards SDL. Francom (2010) agrees that implementation of purposeful teaching and learning activities can promote development of students’ self-direction in learning. Teaching and learning activities aimed at development of SDL skills may contribute to students’ success in their current as well as their future learning (Kan’an & Osman 2015).

The primary components of SDL include, among other things, metacognition (Long & Associates 2000) and effective cooperation with other individuals in a group (Donaghy 2005). Although the outcomes of their research were not related to students’ levels of SDL, but rather to mathematics achievement, the extended research of Kramarski (2004), Kramarski (2008), Mevarech and Fridkin (2006) and Hogan et al. (2015) underscores the positive influence of purposeful development of metacognitive skills in problem-solving contexts, which is important for the context of this study. Similar positive outcomes were reached with the research of Kramarski (2004) and Bernard and Bachu (2015) relating to metacognitive instruction while students were working in groups. Conscious application of metacognitive strategies during cooperative problem-solving evokes consideration and comparison of the advantages and disadvantages of different problem-solving strategies (Kramarski, Mevarech & Arami 2002). It must be noted that the ability to choose and implement appropriate learning strategies or, as in this case, problem-solving strategies is one of the characteristic features of the self-directed learner (Williamson 2007).

The importance of educators implementing teaching and learning strategies that promote metacognitive development in collaborative learning environments is also emphasised by researchers, such as Bernard and Bachu (2015) and Williams and Upchurch (2001) in their research involving students who were solving problems by programming in pairs. Joseph (2009) specifically recommends development of metacognitive thinking strategies during collaborative problem-solving. Research on explicit metacognitive support and development when students are working in groups generally reports positive results in terms of academic achievement, but there seems to be a lack of research specifically on the influence of a teaching and learning strategy that promotes a metacognitive approach to pair problem-solving (PPS) on students’ levels of SDL.

The purpose of the research, which this article reports on, was to explore the influence of a metacognitive approach to PPS on students’ levels of SDL. The research question was ‘How are students’ levels of SDL influenced by using metacognitive self-questions (MSQs) when solving problems in cooperative pairs?’ The article is structured as follows. Initially, an overview is given of the conceptual-theoretical framework on which the empirical investigation was based. Thereafter, the methodology that was employed is discussed, findings are considered and recommendations are made. The article concludes with some final comments.

Conceptual-theoretical framework

The aim of education should not only be to increase students’ knowledge of specific subject content, but also to support them in developing skills that might be beneficial beyond the specific course content (Dynan et al. 2008). The information explosion of the last few decades has caused a significant change in the process of learning, in the sense that special learning skills are needed to adapt to and survive the loads of available information (Saks & Leijen 2014). For students to be academically and otherwise successful to their fullest potential, it has become essential that they engage in SDL (Kan’an & Osman 2015).

Self-directed learning

The increased focus on SDL has led to extensive consideration of the concept since the 1970s (Gassner 2009; Saks & Leijen 2014). Song and Hill (2007) point out that two schools of thought have developed over time: one focusing on the process of SDL (Harrison 1978; Knowles 1975; Mocker & Spear 1982; Oswald 2003) and the other focusing on the characteristics of self-directed learners (Garrison 1997; Guglielmino 1978; Kasworm 1988). Brockett and Hiemstra (1991) distinguish between ‘self-directed learning’ as the instructional process and ‘learner self-direction’ as the characteristics of the learner. They suggest that ‘self-direction in learning’ includes both the external characteristics of the instructional process and the internal characteristics of the learner (Brockett & Hiemstra 1991).

Based on the different perspectives of SDL, a number of definitions have emerged, which have been influenced by various aspects (Dehnad et al. 2014). Self-directed learning has been viewed from the context of educational settings, workforce productivity or learning in the broader society (Bolhuis 2003). For example, from the viewpoint of SDL as a process, Knowles (1975) indicates that SDL occurs when individuals take initiative and responsibility – individually or in collaboration with others – identifying learning needs, formulating learning goals, finding appropriate learning resources, implementing appropriate strategies to reach their goals and evaluating the outcomes of the process. By contrast, as an example of the viewpoint of SDL as a set of personal attributes, Guglielmino (1978) lists the characteristics of a self-directed learner as the following: initiative, independence and persistence in learning; the ability to accept responsibility for own learning and to view problems as challenges, not as obstacles; self-discipline and curiosity; a desire to learn, and self-confidence; the ability to manage own learning; enjoyment of learning and goal orientation. More recently, Saks and Leijen (2014:198) succinctly described a SDL as being ‘able to decide what needs to be learned next and how his learning is best accomplished’. Commonalities in the different perspectives of SDL are that a self-directed learner takes initiative in the learning experience, accepts responsibility for formulating and achieving learning goals, has the ability to identify appropriate resources and learning strategies, learns independently, learns from and with others and regulates their own learning. These commonalities are encompassed in the view of SDL recognised in this article.

One’s propensity to grow into a SDL largely depends on intrinsic motivation, that is, the ability to perform tasks for your own sake and because you enjoy doing so (Guglielmino 2013). The ability to self-direct one’s learning is present in each person to some degree (Guglielmino 1978), but students differ in their levels of intrinsic motivation (Cherry 2017), and therefore also in their levels of SDL (Kan’an & Osman 2015). Intrinsic motivation is the basis of self-determination theory (Gagné & Deci 2005). Self-determination theory (Deci & Ryan 2002; Wehmeyer et al. 2017) suggests that individuals have the innate and universal need for competence, connectedness (relatedness) and autonomy. Self-directed learning affords one opportunity to gain mastery of tasks and learn new skills (competence), to interact with others and feel accepted and connected (relatedness) and to experience the feeling of being in control of one’s own goals, decisions, choices and actions (autonomy) (Guglielmino 2013). When individuals experience a sense of competence, connectedness and autonomy, they become intrinsically motivated to engage in activities that foster personal growth (Cherry 2017). However, such growth requires continuous sustenance, where social support is a key aspect (Deci & Ryan 2002).

Educators’ involvement in encouraging and intentionally developing their students’ SDL skills, and in sustaining their students’ personal growth, is of utmost importance. Learning activities that promote the development of SDL skills may be to the benefit of students to be successful in their current and future learning (Kan’an & Osman 2015). According to Gureckis and Markant (2012), people tend to be more successful learners when they control the learning process themselves, which means that they have the ability to self-direct their learning. Educational scholarship has shown that educators can promote development of SDL skills and independent learning through purposeful implementation of relevant activities and strategies in teaching and learning (Francom 2010; Thornton 2010). Such activities and strategies include, among other things, problem-solving and application of metacognitive strategies (Havenga et al. 2013). A metacognitive approach enables students to manage their learning, which leads them to become responsible and independent learners (Schraw & Dennison 1994), that is, more self-directed learners.

Development of metacognitive awareness and skills

Becoming a self-directed learner necessitates a metacognitive awareness (MA) of one’s own learning processes (Gassner 2009). Metacognitive awareness involves knowledge of how one constructs knowledge and develops understanding, knowledge of various learning strategies, knowledge of the demands of different learning tasks and the skills to plan, monitor and evaluate one’s own learning (Anderson & Nashon 2007). A lack of MA and skills makes it difficult to independently take control of and manage one’s own learning, that is, to move towards SDL. Supporting students to become successful self-directed learners may thus be enhanced through stimulating their metacognitive development (Cotterall & Murray 2009). Application of metacognitive skills can change a routine learning process into a learning experience, and it is therefore considered to be important for lifelong learning in all ages and phases of life (Eldar & Miedijensky 2015; Moseley et al. 2005).

Most students need to be taught and encouraged to develop metacognitive skills (Joseph 2009). It seems that most students are ignorant about metacognition and are not involved in metacognitive thinking, unless they are encouraged to do so through appropriate teaching and learning activities (Bransford, Brown & Cocking 2000; Gezer-Templeton et al. 2017). Metacognition may automatically increase with age, but appropriate learning opportunities can accelerate metacognitive development and enhance subsequent learning (Tan, Biswas & Schwartz 2006). Educators have a responsibility to help their students develop metacognitive skills and apply them in appropriate learning situations (Peters 2000). To the extent to which students make progress in using metacognitive skills, their ability to work and learn in a constructivist perspective increases. Their ability to evaluate their own knowledge construction increases and becomes steady, leading to a greater degree of independence in learning and problem-solving (Peters 2000).

Promotion of metacognition in the classroom starts by making students aware of what metacognition and its contribution to academic success are (Schraw 2001). Students who know the benefits of metacognitive strategies and how to apply them display a greater ability to transfer knowledge and skills acquired in other learning situations (Bannert, Hildebrand & Mengelkamp 2009). Students’ personal awareness of the available learning strategies and the conditions under which these strategies are applied differ (Anderson & Nashon 2007). A variety of learning strategies should therefore be presented and discussed, and students should then be guided to construct explicit knowledge about how, when and where to use the strategies (Ormand 2011; Schraw 2001; Schraw & Gutierrez 2015; Wilson & Johnson 2000). Finally, provision must be made for students to apply and practise these new skills (Ormand 2011).

The metacognitive skills needed to carry out SDL activities include planning, monitoring and evaluating (Van Merriënboer & Sluijsmans 2009). Planning involves setting goals and selecting appropriate strategies to attain these goals, monitoring involves keeping track of one’s progress towards attaining the set goals and evaluating involves assessment of both the product and the process, and the extent to which the goals have been achieved (Van Merriënboer & Sluijsmans 2009). As such, students take responsibility over the whole learning cycle, which should contribute to increasing their levels of SDL.

Self-directed learning and cooperative pair problem-solving

Since the first introduction of the concept of SDL, researchers and authors have addressed the role of collaboration in SDL. Knowles (1975) opined that SDL can occur successfully with or without peer or facilitator collaboration, which was supported by the instructional model of Long (1989), who viewed collaboration during SDL as possible but not necessary. Other, and especially more recent, literature regards collaboration as essential in SDL, for example, Brockett and Hiemstra (1991), Garrison (1997), Oswald (2003) and Strods (2010). Silén and Uhlin (2008) suggest that students, in their development towards higher levels of SDL, need to collaborate with and get feedback from their educators, while peers can be regarded as resources, models and guides for self-direction in learning (Lee et al. 2014; Oswald 2003). Supporting students to become successful self-directed learners may thus be enhanced by implementing collaborative learning activities (Peters & Gray 2005; Strods 2010).

Cooperative PPS, a form of collaboration implemented in this research, is grounded on the principles of pair programming (PP) (Williams & Upchurch 2001), with incorporation of the five basic elements of cooperative learning (Johnson & Johnson 2013). A social-constructivist context is created, where learning occurs as a result of the interaction between the two members of the pair working together at the same computer to solve a problem. In the research that this article reports on, the problem-solving involved advanced competence in using Microsoft Excel. With PPS (as in this research), each group member fulfils a specific role. The driver is actively involved in typing the solution on the computer (Nicolescu & Plummer 2003; Salleh, Mendes & Grundy 2012), while the navigator is responsible for guiding the problem-solving process, by suggesting strategies, consulting resources, observing the driver’s work and making proposals to improve the solution (Berenson et al. 2004; Salleh et al. 2012). Regular switching of roles ensures that both students get the opportunity to develop the different skills.

In cooperative PPS, the notion of social interdependence is strengthened by incorporating the five basic elements generally acknowledged as being essential for cooperative learning to be effective, as proposed by Johnson and Johnson (2013). The elements are: positive interdependence (group members perceive that no one can succeed unless every other member of the group succeeds), individual accountability (every group member’s individual responsibility to successfully complete their own share of the group activity), promotive face-to-face interaction (group members encourage and facilitate each other’s efforts to achieve the goals), appropriate social skills (active listening, conflict management, good communication, leadership, decision-making and trust building) and group processing (self-evaluation of group members’ cooperation, and how well they have achieved their goals). Mentz, Van der Walt and Goosen (2008) were the first to introduce inclusion of cooperative learning principles in PP – a strategy that proved to increase the success of PP considerably.

Cooperative PPS not only affords students’ opportunity to learn from one another but may also promote students’ needs in terms of self-determination theory, as embodied in SDL. Fostering of positive interdependence among group members addresses the need for connectedness (relatedness); fostering individual accountability addresses the need for autonomy; doing group processing to evaluate and realise one’s success addresses the need for competence.

Empirical investigation

Research design

Following a pragmatic approach, both quantitative and qualitative methods were implemented. According to Ivankova, Creswell and Plano Clark (2007), a pragmatic approach aims at better understanding of different aspects of the same problem. The quantitative part of the research entailed a one-group pre-test–post-test experiment (Babbie 2007; Leedy & Ormrod 2005) to determine how students’ levels of SDL and MA were influenced by implementation of a set of MSQs when doing cooperative PPS tasks. A basic qualitative approach (Merriam 2009) was followed in the qualitative part of the research, which entailed individual interviews related to the students’ experiences and views of using the MSQs during problem-solving in cooperative pairs, and how it influenced their self-directedness in learning.


The research involved a class of second-year South African students (n = 33) who were being trained as secondary level computer applications technology (CAT) teachers. This sample was purposeful and convenient, because one of the authors was the lecturer for the specific module where the intervention was implemented. Pre- and post-test questionnaires related to both SDL and MA were completed by 29 students, of which 25 were female students and four were male students, all in the age group of 20–21 years. Four students missed either the pre- or post-test and were therefore not considered in the data analyses. The participants were from different race groups, with only one not having full command of the language of teaching and learning, but she was accommodated through provision of translation into English.


The overall approach of the lecturer in this module over one semester was implementation of cooperative PPS. At the start of every cooperative PPS session, the students were randomly divided into pairs. If an uneven number of students attended class, one of them worked individually. When doing cooperative PPS tasks, the lecturer also included the use of MSQs to prompt students to move beyond mere cognition. These questions were provided in electronic format as a checklist and had to be attended to cooperatively in each of the three phases (planning, practical implementation and evaluation) of the task. In the planning phase, while analysing the problem and designing a solution, the pair had to answer questions that were intended to direct their metacognitive activities in this phase, for example, ‘Do we understand exactly what outcomes we have to reach with the solution of this problem?’ and ‘Do we know strategies/procedures that are appropriate to solve this problem?’ During the practical implementation of their planned solution, the pair had to answer MSQs such as ‘Do we continuously ensure that we understand what we are doing?’, ‘Do we think about how each step fits into what has already been done and what needs to be done next?’ and ‘Do we monitor the progress made with the practical implementation?’ Examples of MSQs that the pair had to answer in the evaluation phase of the problem-solving process include ‘Have we thought about whether the best possible strategies/procedures were used to solve the problem?’ and ‘Did we check if the expected outcomes had been reached?’

Data collection and instrumentation

For the pre-test and the post-test of the quantitative data collection, Williamson’s (2007) self-rating scale of self-directed learning (SRSSDL) and an adapted version of Schraw and Dennison’s (1994) metacognitive awareness inventory (MAI) were used.

Williamson’s (2007) self-rating scale of self-directed learning

Williamson (2007) developed the SRSSDL for use in higher education to assess SDL behaviour as an indication of the level of self-directedness in learning. The questionnaire consists of 60 items, which are categorised into the five broad areas (constructs) of SDL, namely awareness (of factors contributing to becoming a self-directed learner), learning strategies (which need to be adopted to become self-directed in learning), learning activities (which one needs to engage in to become self-directed in learning), evaluation (which is needed to monitor own learning) and interpersonal skills (which are prerequisites to becoming a self-directed learner). Each area includes 12 items. Responses are rated on a five-point Likert scale, where 1 represents ‘never’, and 5 represents ‘always’. The maximum possible total for the questionnaire is 300. Individual scores from 60 to 140 represent a low (L) level of SDL, scores from 141 to 220 represent a moderate (M) level and scores from 221 to 300 represent a high (H) level (Williamson 2007). The reliability of the questionnaire as a whole, as well as the five categories, was confirmed by acceptable Cronbach’s alpha coefficient values (Williamson 2007). The questionnaire is used frequently in SDL research in South Africa. The validity of the questionnaire for the South African context has been established by researchers such as Golightly and Brockett (2010). In this research, the reliability of the questionnaire confirms the reliability of Williamson’s (2007) SRSSDL.

Adapted version of Schraw and Dennison’s (1994) metacognitive awareness inventory

The purpose of this questionnaire was to determine the participants’ self-reported MA, and it was based on Schraw and Dennison’s (1994) MAI. The questionnaire was adapted to make it relevant for PPS. In accordance with the MAI, the questionnaire used in this research also consisted of 52 items, which were divided into two main components, with eight categories (constructs). The main component metacognitive knowledge includes the categories declarative knowledge, procedural knowledge and conditional knowledge. The main component metacognitive control includes the categories planning, information management strategies, monitoring, debugging strategies and evaluation. Responses were rated on a six-point Likert scale, where 1 represents ‘not at all true of me’, and 6 represents ‘always true of me’. The Cronbach’s alpha values for these categories were all higher than 0.7, thus confirming the reliability of the categories for the specific population.

At the end of the intervention period, seven randomly- selected participants were individually interviewed on their views and experiences of using MSQs during problem-solving in cooperative pairs. The first question of the interview had five sub-questions focusing on the abilities of a self-directed learner, according to Knowles’ (1975) definition of SDL. The question that the participants were to respond to was ‘How did implementation of the PPS and the MSQs in this class influence your ability to:

  • identify your own learning needs (what you want or need to learn)? Explain your answer.
  • formulate your own learning goals? Explain your answer.
  • identify human and material resources for learning? Explain your answer.
  • choose and implement appropriate learning strategies? Explain your answer.
  • evaluate the extent to which you have reached the learning goals that you formulated for yourself? Explain your answer’.

The second interview question focused on the use of the MSQ, ‘Please describe your experience of using the MSQs during your cooperative PPS activities in this class’.

Data-processing procedures

Processing of the quantitative data included, firstly, statistical techniques and methods to determine the reliability of all constructs in the two questionnaires, by computing the Cronbach’s alpha values (Anastasi 1988), as reported in the previous section. Paired-samples t-tests were performed to determine statistically-significant (p) and practically-significant differences (d) between the categories of the questionnaires.

A basic qualitative analysis approach was followed in the analysis of the qualitative data. Transcription of the individual interviews was followed by utilising the computer software Atlas.ti to identify relevant segments of text. These segments were linked to codes, categories and themes that emerged from the data. Identification of the themes was guided by the literature and the theoretical framework used in this article. Reliability of the processes was ascertained by having transcripts checked for correctness, and codes, categories and themes cross-checked by another expert in the field of study.

Ethical considerations

Ethical clearance for the larger SDL project, of which this research is part, was obtained from the relevant research ethics committee. Informed consent was obtained from the students, and they were informed that they had the right not to complete the questionnaires or to participate in the individual interviews, without incurring any consequences. However, all the students experienced the intervention, as this intervention was the lecturer’s chosen teaching-learning strategy. The participants were assured that all data would be treated confidentially. Data collection occurred during class time, which did not cause inconvenience to the participants.

Results and findings

To explore the findings that address the aim of the research, results and findings will be reported under three main headings.

Students’ levels of metacognitive awareness before and after the intervention

Table 1 illustrates the results from both the pre-test and the post-test of the MAI.

TABLE 1: Metacognitive awareness inventory – Pre-test and post-test.

From Table 1, it is evident that students’ metacognitive knowledge and metacognitive control had increased with both statistical and practical significance. Regarding the specific constructs, one sees a large practically significant change in the procedural knowledge, planning and monitoring constructs, although all other constructs had a medium practically significant difference, except for debugging (which had a small practically significant difference).

It is reasonable to conclude that the intervention made a difference to students’ MA. It is not surprising that students excelled in the planning and monitoring constructs, as these activities had been deliberately stimulated in class. The lecturer required students to actively plan their solutions and afterwards complete the MSQs that required them to deliberately evaluate their solutions. One could speculate that it is because of the deliberate focus on metacognition that students’ monitoring increased significantly, possibly because of the MSQs becoming an instinctive activity, rather than a deliberate conscious activity. A more detailed discussion of these results is presented in the ‘Discussion’ section.

Students’ self-directedness before and after the intervention

To establish students’ levels of self-directedness in learning, the SRSSDL was implemented before and after the intervention. Table 2 illustrates the results from the pre-test and the post-test for the SRSSDL.

TABLE 2: Self-rating scale of self-directed learning – Pre-test and post-test.

From Table 2, it is evident that no statistically-significant differences were observed, and only small practically significant differences were observed. As the SRSSDL total did not present a large practically-significant difference between the pre-test and the post-test for the whole group, we differentiated the results by dividing the results into two groups, based on participants’ individual scores. Group 1 was the participants with individual scores between 60 and 220, while group 2 was the participants with individual scores between 221 and 300. Table 3 presents the results for group 1, and Table 4 presents the results for group 2.

TABLE 3: Self-rating scale of self-directed learning – Pre-test and post-test (Group 1).
TABLE 4: Self-rating scale of self-directed learning – Pre-test and post-test (Group 2).

From Table 3, it is evident that students who scored 220 or lower in the pre-test of the SRSSDL showed improvement in self-directedness, as is evident from the medium and large practically significant differences observed in four of the five constructs. The practically significant difference (d = 1.20) for the SRSSDL total needs to be noted. The learning activities and evaluation constructs show the largest differences. A statistically significant difference is observed for the evaluation construct.

Table 4 shows that the students who had scored higher than 220 in the pre-test of the SRSSDL did not show a significant increase in self-directedness. This is, however, not of concern, as it is more advantageous to increase the self-directedness of students who have lower levels of SDL. However, a medium practically significant difference is observed in the interpersonal skills construct.

The quantitative findings reported in these two sections clearly indicate that students in this research showed an increase in metacognition, as well as in some of the broad areas of SDL. The following section will report on the qualitative findings from the individual interviews with the students.

Students’ experience of metacognitive self-questions during pair problem-solving

To understand students’ experience of the MSQs used during PPS, seven randomly-selected students were asked to participate in semi-structured individual interviews at the end of the module. Participants 1 and 4 were male students and the other five were female students. In this section, the results from these interviews are reported on. The seven interviewees are indicated with P1, P2, et cetera.

From Table 5, it is evident that three main themes emerged: cooperative learning, SDL and metacognition. In this article, we report only on the SDL and metacognition aspects, although it is clear that the intervention also influenced the students’ view of cooperative learning.

TABLE 5: Codes, categories and themes identified from students’ semi-structured interviews.
Self-directed learning

Several students commented on how the MSQs used in the PPS had become a useful resource and that they enjoyed this way of learning (the PPS). For instance, one student stated: ‘I really enjoyed the CAT [module] this year […] I would not change anything’ (P1, male, 21 years old). Another student commented on the fact that they understood the work better, which shows that they had experienced learning success: ‘At the end the exam was much easier than the previous times, not because the work was easier, but because I understood the work better’ (P3, female, 20 years old). Students also expressed their enhanced motivation for learning in terms of aspects such as interest, enjoyment and goal setting. One of the primary aspects of SDL is taking responsibility for your own learning. The majority of students mentioned how they had learnt to take responsibility for their own learning:

‘[…] if you did not know something yesterday, and you show up the next day and you still do not know, they [the person you are working with] will notice that you are not putting in an effort, or you are not interested’. (P1, male, 21 years old)

Another student stated: ‘You feel responsible for the other person’s marks’ (P7, female, 20 years old), which indicates that marks are still a driving force for learning. Although it may seem that fear of low achievement could be driving these students, it is still evident that they are encouraged to take more responsibility for their own learning.

These findings from the individual interviews indicate how use of MSQs influenced the students’ self-directedness in learning in terms of use of resources, experiencing learning success, motivation and taking responsibility for learning. These findings are related to the findings derived from the SRSSDL. Being more motivated (which includes interest, enjoyment and goal setting) and taking more responsibility for own learning (which includes individual responsibility and positive interdependence) are related to one’s awareness of factors contributing to SDL. Likewise, identification of appropriate resources and the activities resulting in learning success are related to learning strategies that need to be adopted to become self-directed in learning.


As this investigation deliberately focused on fostering metacognition by implementing metacognitive self-questioning in PPS, students were specifically asked to comment on their experience of the MSQs. The majority of students stated that they found the MSQs valuable. For instance, one student was elated when asked about the value of the MSQs:

‘Oh yes, it [the metacognitive self-questions] really helped a lot, to see what I did and what I didn’t do, and if I had forgotten something, to look at that next time’. (P6, female, 20 years old)

Another positive prospect noted by several students was the fact that the MSQs can be applied in other modules too – this leads to knowledge transfer:

‘[…] it really is something that you can use in all the modules. Now I have a template of it [the metacognitive self-questions], so I will definitely use it in my other modules too’. (P2, female, 21 years old)

Critical thinking skills require a strong sense of metacognition. It was therefore not surprising that student’s made mention of aspects related to critical thinking: ‘[…] there were a lot of things that I did not know before that I [now] learnt from other ways of thinking’ (P4, male, 20 years old). Although metacognition was deliberately implemented, some students stated that they felt that the MSQs had become an automatic venture:

‘[…] now and then you think “What now?” again, and then you quickly have a look [at the metacognitive self-questions], so it sinks in, but I think it is because of the repetition’. (P3, female, 20 years old)

These findings from the individual interviews indicate how the use of MSQs influenced the students’ MA in terms of transfer of knowledge and skills, critical thinking and automatic implementation of a metacognitive approach to learning. Greater ability to transfer knowledge and skills and to think critically relates to enhanced procedural knowledge, planning and monitoring, which were the constructs of the MAI that yielded significant differences.

Negative experiences

As with all teaching strategies, students did not all have positive experiences throughout the whole module. The most frequently cited negative experience was centred on the fact that students wanted to work on the computers themselves, and not alternately with the other member of the pair. It needs to be emphasised though that no negative experience related to the MSQs was cited.


The rationale for implementing the specific teaching-learning strategy that this article reports on is based on empirical evidence of the role of metacognition in problem-solving (Christoph 2006; Havenga 2015; Joseph 2009) and SDL (Cotterall & Murray 2009; Gassner 2009; Long & Associates 2000), as well as the role of cooperative learning in SDL (Breed 2016; Mentz & Van Zyl 2016; Oswald 2003; Strods 2010). This research aimed to explore the influence of using MSQs during PPS on students’ levels of SDL. What was observed was that the use of MSQs not only positively influenced the MA of the whole group, but it also positively influenced the self-directedness in learning of students who initially scored low or moderate levels of SDL, as defined by Williamson (2007).

The findings of this research provide evidence that the use of MSQs during PPS firstly enhanced the students’ MA. Both statistically significant and large practically significant differences were found, specifically regarding procedural knowledge, strategies related to planning and monitoring and metacognitive control in general. Although there is a lack of evidence specifically on the influence of using MSQs during PPS in Excel on students’ levels of SDL, the findings of this research are in line with previous research on metacognitive teaching–learning approaches in problem-solving contexts, for example, in mathematics (e.g. Gama 2004; Kramarski 2008; Mevarech & Fridkin 2006), which produced evidence of improvement of MA and skills, which, in turn, may lead to higher levels of SDL (Gassner 2009). The research again underscored the fact that most students need to be purposefully taught and encouraged to develop metacognitive skills (Joseph 2009), as was performed through the specific teaching–learning strategy implemented in this research. According to Kramarski et al. (2002), purposeful development of metacognitive skills when students are working in groups promotes identification of different appropriate strategies to solve a problem, comparison of different strategies and analysis of the advantages and disadvantages of each strategy. These were the activities expected of the two members during PPS in Excel.

During this research, it became evident that students who initially scored low or moderate levels of SDL benefited the most from using MSQs during PPS, thus corroborating the finding of Shannon (2008) that students can become successful self-directed learners if their metacognitive development is stimulated. The quantitative results show medium and large practically significant differences in students’ awareness of factors contributing to becoming self-directed in learning, learning strategies that need to be adopted, learning activities that one needs to engage in and evaluation that is needed for monitoring own learning. These students’ learning motivation increased because of the teaching–learning strategy that stimulated their interest in and their enjoyment of working in pairs. This finding is in line with the research of Liebenberg, Mentz and Breed (2012), who also found that working in pairs increased enjoyment of a task. Using MSQs as a resource also required on the part of the students that they set their own learning goals, which, along with their increased motivation, positively influenced their levels of taking responsibility for own and others’ learning and their awareness of factors contributing to becoming self-directed in learning.

An important finding emanating from the interviews is the students’ experience of learning success because of the use of MSQs, their involvement in selecting appropriate strategies to solve problems and their feeling of better understanding. The use of MSQs influenced their realisation that being self-directed in learning requires engagement in specific learning activities. The most important outcome of using MSQs is probably development of students’ ability to plan, monitor and evaluate their own learning. Using MSQs during PPS required of the students to plan a solution for a problem, to monitor the implementation and progress thereof and to evaluate the product and process. This continuous reflection on students’ cognitive processes during problem-solving may have instilled a disposition focused on reflection and monitoring of their learning processing, which is reflected by the statistically significant and large practically significant difference yielded by the evaluation construct of the SRSSDL.

The teaching–learning strategy implemented during this research afforded students the opportunity to grow in their sense of self-determination. Using MSQs during PPS created opportunities for relatedness (interacting with another student in a pair and feeling connected through mutual responsibility), autonomy (having to set own goals, make decisions and choices and take actions during problem-solving) and competence (mastering tasks and new skills, such as the ability to monitor own learning).

Concluding remarks

This research aimed to determine the influence of using a set of MSQs during PPS on students’ levels of SDL. From the results, it seems that this approach to problem-solving in Excel contributed to improvement of the students’ MA in general, but specifically in terms of procedural knowledge, planning and monitoring. Most importantly, both quantitative and qualitative results reflected that the approach positively influenced the levels of SDL of those students who initially scored moderate or low levels of self-directedness in learning. These students showed an improvement in their awareness of factors that contribute to SDL, learning strategies that need to be adopted to become a self-directed learner, learning activities that one needs to be engaged in and the skills students need to be able to evaluate and monitor their learning process and progress. Being self-directed in learning, and possessing the associated skills and attributes, is essential for the contemporary student to be able to cope in a world that changes at an exponential rate.

This research suggests that SDL in educational contexts can be promoted through a combination of metacognitive approaches and cooperative learning teaching–learning strategies, such as PPS. It is, however, important that cooperative activities incorporate the five elements of positive interdependence, individual accountability, promotive interaction, appropriate social skills and group processing.


Competing interests

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

Authors’ contributions

R.B. was responsible for implementation of the teaching strategy, data collection and analysis, report of results and critical reading of the article. B.B. was responsible for formulating the problem statement, the conceptual-theoretical framework, and analysis of data, discussion and finalisation of the article.


Anastasi, A., 1988, Psychological testing, 6th edn., Macmillan, New York.

Anderson, D. & Nashon, S., 2007, ‘Predators of knowledge construction: Interpreting students’ metacognition in an amusement park physics program’, Science Education 91(2), 298–320. https://doi.org/10.1002/sce.20176

Babbie, E., 2007, The practice of social research, 11th edn., Thomson Wadsworth, Belmont, CA.

Bannert, M., Hildebrand, M. & Mengelkamp, C., 2009, ‘Effects of a metacognitive support device in learning environments’, Computers in Human Behavior 25(4), 829–835. https://doi.org/10.1016/j.chb.2008.07.002

Berenson, S.B., Slaten, K.M., Williams, L. & Ho, C.-W., 2004, ‘Voices of women in a software engineering course: Reflections on collaboration’, Journal on Educational Resources in Computing (JERIC) 4(1), 1–18. https://doi.org/10.1145/1060071.1060074

Bernard, M. & Bachu, E., 2015, ‘Enhancing the metacognitive skill of novice programmers through collaborative learning’, in A. Pena-Ayala (ed.), Metacognition: Fundaments, applications, and trends, pp. 277–298, Springer, London.

Bolhuis, S., 2003, ‘Towards process-oriented teaching for self-directed lifelong learning: A multidimensional perspective’, Learning and Instruction 13(3), 327–347. https://doi.org/10.1016/S0959-4752(02)00008-7

Bransford, J.D., Brown, A.L. & Cocking, R.R. (eds.), 2000, How people learn: Brain, mind, experience, and school, National Academy Press, Washington DC.

Breed, B., 2016, ‘Exploring a cooperative learning approach to improve self-directed learning in higher education’, Journal for New Generation Sciences 14(3), 1–21.

Brockett, R.G. & Hiemstra, R., 1991, Self-direction in adult learning: Perspectives on theory, research, and practice, Routledge, New York.

Cherry, K., 2017, What is self-determination theory?, viewed 24 October 2017, from https://www.verywell.com/what-is-self-determination-theory-2795387

Christoph, L.H., 2006, ‘The role of metacognitive skills in learning to solve problems’, PhD thesis, University of Amsterdam.

Cotterall, S. & Murray, G., 2009, ‘Enhancing metacognitive knowledge: Structure, affordances and self’, System 37(1), 34–45. https://doi.org/10.1016/j.system.2008.08.003

Deci, E.L. & Ryan, R.M. (eds.), 2002, Handbook of self-determination research, University of Rochester Press, Rochester, NY.

Dehnad, A., Afsharian, F., Hosseini, F., Arabshahi, S.K.S. & Bigdeli, S., 2014, ‘Pursuing a definition of self-directed learning in literature from 2000–2012’, Procedia – Social and Behavioral Sciences 116, 5184–5187. https://doi.org/10.1016/j.sbspro.2014.01.1097

Donaghy, R.C., 2005, ‘Studying self-directed learning: The personal stories of four scholars’, International Journal of Self-Directed Learning 2(2), 1–11.

Dynan, L., Cate, T. & Rhee, K., 2008, ‘The impact of learning structure on students’ readiness for self-directed learning’, Journal of Education for Business 84(2), 96–100. https://doi.org/10.3200/JOEB.84.2.96-100

Eldar, O. & Miedijensky, S., 2015, ‘Designing a metacognitive approach to the professional development of experienced science teachers’, in A. Pena-Ayala (ed.), Metacognition: Fundaments, applications, and trends, pp. 299–322, Springer, London.

Francom, G.M., 2010, ‘Teach me how to learn: Principles for fostering students’ self-directed learning skills’, International Journal of Self-Directed Learning 7(1), 29–44.

Gagné, M. & Deci, E.L., 2005, ‘Self-determination theory and work motivation’, Journal of Organizational Behavior 26(4), 331–362. https://doi.org/10.1002/job.322

Gama, C., 2004, ‘Metacognition in interactive learning environments: The reflection assistant model’, in J.C. Lester, R.M. Vicari & F. Paraguacu (eds.), 7th International Conference on Intelligent Tutoring Systems proceedings, Springer, Berlin, August 30–September 3, pp. 668–677.

Garrison, D.R., 1997, ‘Self-directed learning: Toward a comprehensive model’, Adult Education Quarterly 48(1), 18–33. https://doi.org/10.1177/074171369704800103

Gassner, L., 2009, ‘Developing metacognitive awareness – A modified model of a PBL tutorial’, Bachelor thesis, Malmö University, Sweden, viewed 29 June 2016, from http://dspace.mah.se/handle/UppsatsGassner

Gezer-Templeton, P.G., Mayhew, E.J., Korte, D.S. & Schmidt, S.J., 2017, ‘Use of exam wrappers to enhance students’ metacognitive skills in a large introductory food science and human nutrition course’, Research in Food Science Education 16(1), 28–36. https://doi.org/10.1111/1541-4329.12103

Golightly, A. & Brockett R.G., 2010, ‘Teacher trainees’ perceptions of their own self-directedness in learning’ in D. Frances, S. Mahlomaholo & M. Nkoane (eds.), Praxis towards sustainable empowering learning environments in South Africa, pp. 43–61, Sun Media, Bloemfontein.

Guglielmino, L.M., 1978, ‘Development of the Self-directed Learning Readiness Scale’, PhD dissertation, University of Georgia.

Guglielmino, L.M., 2013, ‘The case for promoting self-directed learning in formal educational institutions’, SA-eDUC Journal 10(2), 1–18.

Gureckis, T.M. & Markant, D.B., 2012, ‘Self-directed learning: A cognitive and computational perspective’, Perspectives on Psychological Science 7(5), 464–481. https://doi.org/10.1177/1745691612454304

Harrison, R., 1978, ‘How to design and conduct self-directed learning experiences’, Group & Organization Management 3(2), 149–167. https://doi.org/10.1177/105960117800300203

Havenga, M., 2015, ‘The role of metacognitive skills in solving object-oriented programming problems: A case study’, TD: The Journal for Transdisciplinary Research in Southern Africa 11(1), 133–147. https://doi.org/10.4102/td.v11i1.36

Havenga, M., Breed, B., Mentz, E., Govender, D., Govender, I., Dignum, F. et al., 2013, ‘Metacognitive and problem-solving activities to promote self-directed learning in computer programming: Teachers’ perspectives’, SA-eDUC Journal 10(2), viewed 24 October 2017, from http://www.saeduc.co.za/index.php/saeduc/article/view/36

Hogan, M.J., Dwyer, C.P., Harney, O.M., Noone, C. & Conway, R.J., 2015, ‘Metacognitive skill development and applied systems science: A framework of metacognitive skills, self-regulatory functions and real-world applications’, in A. Pena-Ayala (ed.), Metacognition: Fundaments, applications, and trends, pp. 75–106, Springer, London.

Ivankova, N.V., Creswell, J.W. & Plano Clark, V.L., 2007, ‘Foundations and approaches to mixed methods research’, in K. Maree (ed.), First steps in research, pp. 253–282, Van Schaik, Pretoria.

Johnson, D.W. & Johnson, F.P., 2013, Joining together: Group theory and group skills, 11th edn., Pearson, Upper Saddle River, NJ.

Joseph, N., 2009, ‘Metacognition needed: Teaching middle and high school students to develop strategic learning skills’, Preventing School Failure: Alternative Education for Children and Youth 54(2), 99–103. https://doi.org/10.1080/10459880903217770

Kan’an, A. & Osman, K., 2015, ‘The relationship between self-directed learning skills and science achievement among Qatari students’, Creative Education 6(8), 790–797. https://doi.org/10.4236/ce.2015.68082

Kasworm, C.E., 1988, ‘Part-time credit learners as full-time workers: The role of self-directed learning in their lives’, paper presented at the meeting of the American Association of Adult and Continuing Education, Tulsa, OK, October.

Knowles, M.S., 1975, Self-directed learning: A guide for learners and teachers, Association Press, New York.

Kramarski, B., 2004, ‘Making sense of graphs: Does metacognitive instruction make a difference on students’ mathematical conceptions and alternative conceptions?’, Learning and Instruction 14(6), 593–619. https://doi.org/10.1016/j.learninstruc.2004.09.003

Kramarski, B., 2008, ‘Promoting teachers’ algebraic reasoning and self-regulation with metacognitive guidance’, Metacognition and Learning 3(2), 83–99. https://doi.org/10.1007/s11409-008-9020-6

Kramarski, B., Mevarech, Z.R. & Arami, M., 2002, ‘The effects of metacognitive instruction on solving mathematical authentic tasks’, Educational Studies in Mathematics 49(2), 225–250. https://doi.org/10.1023/A:1016282811724

Lee, K., Tsai, P.-S., Chai, C.S. & Koh, J.H.L., 2014, ‘Students’ perceptions of self-directed learning and collaborative learning with and without technology’, Journal of Computer Assisted Learning 30(5), 425–437. https://doi.org/10.1111/jcal.12055

Leedy, P.D. & Ormrod, J.E., 2005, Practical research: Planning and design, 8th edn., Prentice-Hall, Upper Saddle River, NJ.

Liebenberg, J., Mentz, E. & Breed, B., 2012, ‘Pair programming and secondary school girls’ enjoyment of programming and the subject Information Technology (IT)’, Computer Science Education 22(3), 219–236. https://doi.org/10.1080/08993408.2012.713180

Long, H.B., 1989, ‘Self-directed learning: Emerging theory and practice’ in H.B. Long & Associates (eds.), Self-directed learning: Emerging theory and practice, pp. 1–12, Oklahoma Research Center for Continuing Professional and Higher Education, University of Oklahoma, Norman, OK.

Long, H.B. & Associates, 2000, Practice and theory in self-directed learning, Motorola University Press, Schaumburg, IL.

Mentz, E., Van der Walt, J.L. & Goosen, L., 2008, ‘The effect of incorporating cooperative learning principles in pair programming for student teachers’, Computer Science Education 18(4), 247–260. https://doi.org/10.1080/08993400802461396

Mentz, E. & Van Zyl, S., 2016, ‘Introducing cooperative learning: Students’ attitudes towards learning and the implications for self-directed learning’, Journal of Education 64, 79–109.

Merriam, S.B., 2009, Qualitative research: A guide to design and implementation, Jossey-Bass, San Francisco, CA.

Mevarech, Z. & Fridkin, S., 2006, ‘The effects of IMPROVE on mathematical knowledge, mathematical reasoning and meta-cognition’, Metacognition and Learning 1(1), 85–97. https://doi.org/10.1007/s11409-006-6584-x

Mocker, D.W. & Spear, G.E., 1982, Lifelong learning: Formal, nonformal, informal, and self-directed, ERIC Clearinghouse on Adult, Career, and Vocational Education, Ohio State University, Columbus, OH.

Moseley, D., Baumfield, V., Elliott, J., Gregson, M., Higgins, S., Miller, J. et al., 2005, Frameworks for thinking: A handbook for teaching and learning, Cambridge University Press, Cambridge.

Nicolescu, R. & Plummer, R., 2003, ‘A pair programming experiment in a large computing course’, Romanian Journal of Information Science and Technology 6(1–2), 199–216.

Ormand, C., 2011, ‘Teaching metacognition. On the cutting edge’, in Professional development of Geoscience faculty, Carleton College, viewed 22 February 2011, from http://serc.carleton.edu/NAGTWorkshops/metacognition/teaching_metacognition.html

Oswald, D., 2003, ‘Instructional-design theory for fostering self-directed learning’, PhD thesis, Indiana University, Bloomington, IN.

Peters, J.M. & Gray, A., 2005, ‘A solitary act one cannot do alone: The self-directed collaborative learner’, International Journal of Self-Directed Learning 2(2), 12–23.

Peters, M., 2000, ‘Does constructivist epistemology have a place in nurse education?’, Journal of Nursing Education 39(4), 166–172.

Rashid, T. & Asghar, H.M., 2016, ‘Technology use, self-directed learning, student engagement and academic performance: Examining the interrelations’, Computers in Human Behavior 63(2016), 604–612. https://doi.org/10.1016/j.chb.2016.05.084

Salleh, N., Mendes, E. & Grundy, J., 2012, ‘Investigating the effects of personality traits on pair programming in a higher education setting through a family of experiments’, Empirical Software Engineering 19, 714–752. https://doi.org/10.1007/s10664-012-9238-4

Saks, K. & Leijen, A., 2014, ‘Distinguishing self-directed and self-regulated learning and measuring them in the e-learning context’, Procedia – Social and Behavioral Sciences 112, 190–198. https://doi.org/10.1016/j.sbspro.2014.01.1155

Schraw, G., 2001, ‘Promoting general metacognitive awareness’, in H.J. Hartman (ed.), Metacognition in learning and instruction: Theory, research and practice, pp. 3–16, Kluwer, Dordrecht.

Schraw, G. & Dennison, R.S., 1994, ‘Assessing metacognitive awareness’, Contemporary Educational Psychology 19(4), 460–475. https://doi.org/10.1006/ceps.1994.1033

Schraw, G. & Gutierrez, A.P., 2015, ‘Metacognitive strategy instruction that highlights the role of monitoring and control processes’, in A. Pena-Ayala (ed.), Metacognition: Fundaments, applications, and trends, pp. 3–16, Springer, London.

Shannon, S.V., 2008, ‘Using metacognitive strategies and learning styles to create self-directed learners’, Institute for Learning Styles Journal 1, 14–28.

Silén, C. & Uhlin, L., 2008, ‘Self-directed learning – A learning issue for students and faculty!’, Teaching in Higher Education 13(4), 461–475. https://doi.org/10.1080/13562510802169756

Song, D. & Bank, C.J., 2016, ‘Motivational factors in self-directed informal learning from online learning resources’, Cogent Education 3(2016), 1–11. https://doi.org/10.1080/2331186X.2016.1205838

Song, L. & Hill, J.R., 2007, ‘A conceptual model for understanding self-directed learning in online environments’, Journal of Interactive Online Learning 6(1), 27–42.

Strods, G., 2010, ‘Promotion of students’ self-directed learning readiness in cooperative learning’, in Education reform in comprehensive school: Education content research and implementation problems, The collection of scientific papers 2010, pp. 120–134, Rezekne Higher Education Institution, Rezekne, Latvia.

Tan, J., Biswas, G. & Schwartz, D.L., 2006, ‘Feedback for metacognitive support in learning by teaching environments’, in 28th Annual Meeting of the Cognitive Science Society Proceedings, Vancouver, Canada, July 26–29, pp. 828–833.

Thornton, K., 2010, ‘Supporting self-directed learning: A framework for teachers’, Language Education in Asia 1(1), 158–170.

Van Merriënboer, J.J.G. & Sluijsmans, D.M.A., 2009, ‘Toward a synthesis of cognitive load theory, four-component instructional design, and self-directed learning’, Educational Psychology Review 21(1), 55–66. https://doi.org/10.1007/s10648-008-9092-5

Wehmeyer, M.L., Shogren, K.A., Little, T.D. & Lopez, S.J., 2017, Development of self-determination through the life-course, Springer, Dordrecht.

Williams, L. & Upchurch, R., 2001, ‘In support of student pair-programming’, ACM SIGCSE Bulletin 33(1), 327–331. https://doi.org/10.1145/366413.364614

Williamson, S.N., 2007, ‘Development of a self-rating scale of self-directed learning’, Nurse Researcher 14(2), 66–83. https://doi.org/10.7748/nr2007.

Wilson, J. & Johnson, P., 2000, ‘Students thinking about their learning: Assessment to improve learning’, Educational Research Quarterly 24(2), 10–20.

Crossref Citations

No related citations found.