Dr Math was a mobile mathematics tutoring service, used by school learners across South Africa. The wealth of historic data available, with regard to the conversations between tutors and learners, may contain valuable insights as to which mathematics topics are most frequently encountered on the Dr Math service. This alignment may serve as an indicator of the utility of an online tutorial service as a reflection of the curriculum covered by learners, and as an extra avenue of support. This study makes use of automated means to rank the topics discussed on the Dr Math service and to align them with the topics encountered in the South African National Senior Certificate final examinations. The study finds that there is a close alignment with regard to the observations of the Department of Basic Education on factors influencing the performance of the learners. The topics most frequently discussed on the Dr Math service also align closely with the topics with which the learners have most difficulty in their final exams.

Dr Math was a mathematics tutorial service accessible on cellular phones. This service allowed South African school learners to get in touch with human mathematics tutors. The tutoring service cost the learner no more than the data charges applicable to using an online chat platform. The mobile nature of the service also allowed the learners to gain access to the tutors from wherever they found themselves at a given moment.

The Dr Math service is, unfortunately, now defunct, but during its operation it stored all of the conversations between the tutors and learners as anonymous text files. These text files contain a wealth of information, to be mined. Using Dr Math and its historic system logs as a case study may reveal subtle alignments between the tutoring process and the real-world educational process.

Dr Math was and is still by no means the only tutoring service available. Although the availability of tutoring services, such as Dr Math, may be seen as a blessing for providing much needed extra help and information to South African learners, it is yet to be determined whether the tutoring help provided by such a service aligns with the South African educational curriculum in any shape or fashion. To that end, this study makes use of Dr Math as a case study.

There are many ways in which both to mine and to interpret the data from the learner–tutor conversations on Dr Math. As the Dr Math service was available to all South African learners, it provided a feasible means of gathering an overview of which mathematical topics are problematic to the school learners in South Africa. Gathering this information would make it possible to determine whether the learners on Dr Math requested help on the same topics with which learners are struggling during examinations. Techniques used in natural language processing (NLP) may be used to discover whether an alignment exists between these topics. If such an alignment exists, then it follows that these online tutoring services align with aspects of the South African mathematics curriculum and, as such, provide a useful extra avenue of support to school learners.

To address this hypothesis, the study proposes to answer a single research question, namely how closely do the mathematical topics that are most frequently discussed on the Dr Math service align with the performance of learners in National Senior Certificate (NSC) Paper I Mathematics examinations? In order to address this issue, it is necessary firstly to determine how learners perform, with regard to specific topics, in their NSC Mathematics final exams. Furthermore, it is necessary to determine which mathematical topics occur most frequently on the Dr Math service.

This study has been conducted in five phases. The first phase constitutes the gathering of background information, by means of literature study. The background information consists of the historic performance data (in the form of rankings) from the South African NSC Mathematics final examinations, as well as information on how the conversations on the Dr Math service may be processed automatically to extract useful information.

This study aims to use the equations found in mathematical textbooks as a means to search through the historic Dr Math logs. By aligning these equations with the chapters and high school grades in which they are found, it is possible to align Dr Math conversations according to mathematical topics. In order to perform this task, several equations need to be processed. Thus, the second phase of the study discusses how the equations were captured and converted to a form that may be used as search terms.

In the third phase of the study, an automated process is used to extract the equations embedded in the conversations on the Dr Math service. These equations are saved in a list, to be aligned with the equations identified in the second phase. The Dr Math conversations are all recorded in anonymous text files which serve as source data for this study. Our study makes use of the Dr Math text logs for the years 2010–2013, which contain 248 993 individual lines of conversation, to provide a historical overview of the equations, and associated topics, discussed on the service.

In the fourth phase of the study, the equations from phases 2 and 3 are aligned. These alignments are used to determine in which grades an equation is encountered, as well as to provide a ranked list of the most frequently encountered topics on the Dr Math service. The final phase of the study attempts to draw an alignment between the NSC rankings created in phase 1 with the Dr Math rankings compiled in phase 4.

To place the conversations on the Dr Math service in context, this section, which serves as the first phase of the study, firstly discusses the performance of learners on the NSC examinations. Furthermore, an overview is provided of what the Dr Math service is as well as the means by which its historic conversations may be processed automatically.

The main goal of this study is to determine whether the topics addressed by the learner–tutor conversations on the Dr Math service reflect the topics with which the South African school learners struggle most on their exams. The South African Department of Basic Education publishes yearly diagnostic reports, which contain a breakdown of the issues faced by learners in the final NSC examination papers of all subjects.

The 2011 NSC examination results for Mathematics report that many of the errors made by learners have their origins in a poor understanding of the basic and foundational mathematical competencies, which have been taught in earlier grades (Department of Basic Education

In their study Mhlolo et al. (

The main underlying conclusions of the 2012 (Department of Basic Education

Paper I topics ranked according to lowest performance.

Ranking | Topic |
---|---|

1 | Calculus |

2 | Functions and graphs |

3 | Linear programming |

4 | Number patterns and sequences |

5 | Annuities and finance |

6 | Algebra, equations and inequalities |

The Dr Math service was devised by the Meraka Institute of the South African Council for Scientific and Industrial Research. Initially, the service was an attempt to see if high school learners would use their own cellular phones to contact mathematical tutors. Eventually, the service grew to accommodate the requests of tens of thousands of school learners.

The tutors on the service were all volunteers, who lent their time freely. The popularity of the service ensured that there were generally many more learners accessing the service than available tutors. This led to availability issues which were partially addressed by providing a queuing system (Butgereit & Botha

The statements in

A few example queries taken from the historic logs of Dr Math.

ok wt dnt u knw in trig

at wht tym wil u be onlyn again cuz nw im buzy eatng? ill be on till five

ok mr i’l get bck 2 u b4 fiv ok cool

…

use our calcluator type in. c 15 - 12 i want u 2 answer

sorry you have to do the dirty work ;-) hv u make sex before like me;)

bye bye

no i want 2 hv sex with u

…

next pr0blem (4p) x (-4p) wait ign0re that

and x is multiply?

the sum is 40ab^2 divided by -5ab are we skipping 4p x -4p?

The conversations on the Dr Math service constitute a form of natural language, called microtext. Microtext may be defined as the short snippets of text used in modern digital forms of communication (Hovy et al.

Natural language processing is a diverse research field, concentrating on both textual and vocal user input. This field pursues the elusive question of how we understand the meaning of a sentence or a document (Feldman

There have been other studies that used NLP techniques to address the problem of mathematics in text. Adeel, Cheung and Khiyal (

Regular expressions are a means to identify valid strings. The strings are matched against a series of patterns. Each individual pattern is referred to as a regular expression. Regular expressions are made up of a series of wildcard and constant characters, examples of which are shown in

Example of regular expression wildcards and constants.

Item | Description |
---|---|

\( | A single opening round bracket |

\) | A single closing round bracket |

[A-Z]{2,6} | A wildcard used to identify two to six capital letters in sequence. These sequences of capital letters represent keywords such as PI and ANGLE. |

[a-zA-Z0-9]+ | A wildcard used to identify at least one numeric or alphabetic character. These sequences represent either numeric constants or alphabetic variants. |

(\+|\-|\*) | A wildcard used to identify a single plus, minus or multiplication operator. These sequences represent mathematical operators. |

\/ | A single division symbol. |

\^ | A single exponent operator. |

[\=|\>|\<] | A wildcard mask used to identify a single assignment or comparison operator. |

The constant characters in the pattern have to match explicitly, whereas the wildcard characters provide a degree of variance. Regular expressions have been used in text processing to remove HTML tags when processing web pages (Li

By using the basis of regular expressions, it is possible to build representative patterns of the equations that may be found in the Dr Math text logs. These patterns would not only be applicable to specific mathematical equations, but would also allow various equations to be identified using the same pattern, thereby extending the applicability of the patterns beyond the equations from which the patterns were initially generated.

A single regular expression and some of the equations to which it could map.

In

As part of the second phase of the study, it was necessary to source example equations as a means of comparison. To facilitate the equation capture, a software application was created to allow any equations from various sources to be captured manually. Notational guidelines were created to facilitate the capture of equations. This ensured that the equations were captured in the same format as equations created by the automated process. Five high school Mathematics textbooks were consulted as sources of equations, each representing a specific grade in high school (Carter et al.

Every captured equation was saved and any incidental spacing removed. For each subsequent entry, the application checked the list of existing equations to ensure that no duplicates were captured. This technique yielded 3145 unique equations. Each of the equations was saved along with details regarding in which high school grade and chapter in the textbook it was first encountered. This study makes use of the chapter names from the textbooks as our general mathematics topics. Some of the chapter names, such as patterns, functions and algebra, are repeated across different grades. Using only the unique chapter names as topics yielded a total of 22 topics.

To provide some idea of the range of equations found in the textbooks and also the format in which they were captured, a few example equations are shown in

Example equations taken from mathematics textbooks.

The software application used to capture the mathematical equations was used to generate a regular expression matching every equation captured from the textbooks, using the wildcards and constants listed in

As part of a related study, an automated system was developed which is capable of processing an input statement from the Dr Math text files to extract any perceived mathematical equation and to structure it in a representative form. The processing of these text files serves as the third phase of this study. The process makes use of various concepts found in the fields of natural language and text processing, not only to extract equations that are explicitly stated, but also those that may be structured in an unexpected manner.

The equations may be structured in unexpected ways as the learners make use of the same forms of language to phrase their questions as they would in chatting with their peers, that is, by leaving out many words or using abbreviated forms. Another influence on the structure of these messages could be that the keypads of most feature (non-smart) mobile phones, which were still used by many school learners during the period 2010–2013, do not lend themselves to entering mathematical equations properly.

Dr Math statements and their associated equations.

Even though the automated process provides a convenient and quick way of processing the Dr Math text logs, the results of the process would only be of use if it could be proven that the process approximates similar judgement to that of a human performing the same translations. To that end, two tests were conducted using human volunteers. The first phase of the tests consisted of having three human volunteers scrutinise 1000 entries found in the Dr Math text logs. They had to perform a simple coding process, stating whether they believed the entry to be translatable or not. Coding is a process in which participants record data according to rules supplied as part of the study. These same entries were processed (coded) by an initial phase of the automated process. This coding process forms part of a simple content analysis task. Content analysis is a research technique for making replicable and valid inferences from texts to the contexts of their use (Krippendorff

The levels of agreement between the three human coders and the automated process was calculated using Krippendorff’s α Krippendorff’s α is a very general measure of intercoder agreement, which allows for uniform reliability standards to be applied to a great diversity of data (Krippendorff

Comparison of calculations between the automated process and individual coders on agreement (α as the first value) and the level of agreement they could have reached by chance (θ as the value in brackets).

A further round of tests was conducted to validate whether the automated process could deliver acceptable translations. To that end, the two human participants who were in relatively high agreement in the first test phase were asked to translate a series of 250 Dr Math statements into mathematical equations. The automated process was tasked to do the same. The equations created by the automated process were compared to those created by the human participants across various metrics. The metric calculations between the two human participants were used as a baseline for the tests.

In all metric calculations, the automated process was able to meet or to exceed the results of the human participants. Furthermore, the metric results calculated between the two human participants showed a moderate level of correlation. This level of correlation was matched by the automated process. These results serve as validation that the automated process is able to identify and to extract equations at a level similar to that which may be expected by a human participant.

To facilitate the tests required for the current study, the statements found in the Dr Math text files were structured according to the year in which they were captured. This provided four separate sets of text files to process, for each of the years 2010 to 2013. The automated process was applied to each statement for a given year. If the process yielded an equation, the equation was saved to a list of equations for the given year.

Statements processed and equations identified per year

Year | Statements processed | Equations identified |
---|---|---|

2010 | 82 289 | 9364 |

2011 | 23 995 | 3141 |

2012 | 61 084 | 9273 |

2013 | 81 625 | 14 363 |

In total, 248 993 unique statements were processed from the Dr Math text files. From these statements, a total of 36 141 equations were extracted. These equations serve as the source data for identifying the mathematical topics encountered on the Dr Math service.

In the final phase of the study, the equations from the second and third phases were aligned by using the regular expressions generated in the second phase to match to the equations generated in the third phase. Regular expressions may be used to form either partial or complete matches to strings. For this study, the regular expressions were used to identify equations generated from the 2010 to 2013 Dr Math text logs with which they formed a complete match. A specific equation may be aligned with various topics found in multiple grades. Although, during processing, all the possible topic associations of an equation were saved, this phase of the study only makes use of the lowest grade (and topic) with which an equation was aligned. This was performed in an attempt to illustrate what the fundamental principles are that are discussed during tutorial sessions.

Number of equations matched to the 22 topics for the period 2010–2013.

Ranking | Module | Matching equations |
---|---|---|

1 | Patterns, functions and algebra | 6764 |

2 | Numbers, operations and relationships | 4765 |

3 | Sequences and series | 1018 |

4 | Solving and graphing nonlinear equations | 469 |

5 | Solving and graphing linear equations | 338 |

6 | Relations and functions | 322 |

7 | The derivative | 254 |

8 | Money matters | 245 |

9 | Inverse relations and functions | 155 |

10 | Numbers and number patterns | 153 |

11 | Measurement | 128 |

12 | Quadratic equations | 123 |

13 | Number patterns | 71 |

14 | Manipulating algebraic expressions | 65 |

15 | Number basics | 52 |

16 | Coordinate geometry | 37 |

17 | The remainder and factor theorems | 22 |

18 | Space and shape | 18 |

19 | Graphs of some other nonlinear functions | 7 |

20 | Trigonometry | 7 |

21 | Linear programming | 5 |

22 | Surface area and volume | 1 |

From these alignments the chart shown in

Distribution of where equations are first encountered by grade for the years 2010–2013.

These results may be interpreted in two ways. The first interpretation is that most of the learners seeking guidance from the Dr Math tutors are in the lower high school grades, namely 8 and 9. The second interpretation is that the results are representative of the fact that most of the learners struggle with concepts of which the basic structure was covered in Grades 8 and 9 and that all other concepts are related to these concepts. This interpretation aligns with the observations made in the diagnostics reports created by the South African Department of Basic Education, as discussed above, that most of the learners who wrote the NSC examinations struggle with the basic concepts taught in Grades 8 and 9.

To further investigate this phenomenon,

The 10 most frequently encountered topics (as measured by the number of equations) for the years 2010 to 2013.

Across the data sets for all four years, the two most frequently encountered topics are those of patterns, functions and algebra, and numbers, operations and relationships. The topic of patterns, functions and algebra deals with such basic concepts as factors and exponents, whereas numbers, operations and relationships address the simplification of algebraic expressions, roots and fractions. As these topics are fairly basic, it further supports our prior theory that they are flagged so frequently simply because their base understanding is a requirement for other mathematical topics.

For the first round of tests, the distributions and rankings were calculated according to the lowest possible grade in which a concept was encountered. From the tests it became clear that most of the concepts aligned with the basic concepts covered in Grades 8 and 9. For the second set of tests, it was necessary to determine whether some of the equations could also match to regular expressions representing higher level concepts. An example of this is the equation 3x − 2y = 5 which may map to equations linked to the topics of patterns, functions and algebra, encountered in Grade 8, and solving and graphing linear equations, encountered in Grade 10, in the textbooks.

To do this the data sets were reprocessed to include all possible matches for an equation on any grade or topic, but the equations matched to the topics of patterns, functions and algebra, and numbers, operations and relationships were relegated only to those instances when the equations could only be aligned to these specific two topics. From these alignments the chart shown in

Distribution of equations by grade on all possible alignments for the years 2010–2013.

With these changes made it is apparent that, based on the equations, the learner queries are distributed fairly evenly between concepts first encountered in Grades 8, 10 and 12. This is to be expected as Grade 8 constitutes the initial contact of the learners with high school-level mathematics topics. New topics are introduced in Grade 10, as this is when the learners make their subject choices for the remainder of their high school career. Finally, Grade 12 represents the outcome-level for high school learning. As such, it is to be expected that a fair number of learners would require help in preparing for their final exams.

For the final comparison, and the final phase of the study, the 22 topics used to distinguish the Dr Math conversations were summarised into the 6 topics identified on the NSC Paper I examinations. Furthermore, for this comparison, only the results from the 2012 and 2013 Dr Math text files were used. This was performed in order to perform a direct comparison with the NSC Paper I topic rankings for 2012 and 2013. The comparison between these rankings and the same topic rankings from the conversations in the Dr Math text logs are shown in

Ranked topics from the NSC examinations compared with the ranked topics from Dr Math.

Ranking | NSC | Dr Math |
---|---|---|

1 | Calculus | Calculus |

2 | Functions and graphs | Functions and graphs |

3 | Linear programming | Number patterns and sequences |

4 | Number patterns and sequences | Linear programming |

5 | Annuities and finance | Algebra, equations and inequalities |

6 | Algebra, equations and inequalities | Annuities and finance |

The NSC rankings indicate which topics learners received the lowest marks for on average in the examinations, whereas the Dr Math rankings indicate which topics were most frequently encountered. There is a definite alignment between the first two ranked topics of calculus, and functions and graphs. Topics 3 and 4 are reversed in order between the NSC and Dr Math results. The same can be said for topics 5 and 6. This further demonstrates the closeness in rankings, as topics are never more than a simple swap away from being aligned.

This study set out to answer the question: How closely do the mathematical topics that are most frequently discussed on the Dr Math service align with the performance of learners in NSC Paper I Mathematics examinations? In order to answer this question, two objectives were addressed.

Firstly, the historic diagnostic reports from the South African Department of Basic Education were consulted, to determine which issues were most pressing. From these reports, the average performance of the students on specific questions (and their associated topics) was used to create a ranked list of six topics in the order in which the learners performed the worst.

Secondly, the historic text logs of Dr Math were processed by automated means to determine which topics occur most frequently. Initially, it was observed that the learners struggled with the basic mathematical concepts covered in Grades 8 and 9. This aligns with the observations made by the Department of Basic Education. The Department of Basic Education lists a lack of exercise on the topics, beyond that which may be found in textbooks and prior question papers, as one of the reasons that the students have such a low level of conceptual understanding of the topics.

Further processing of the data also revealed that the topics most frequently discussed on the Dr Math service aligned relatively closely with the topics which the learners found most problematic on their NSC final examinations. This serves to answer the research question of the study. Furthermore, the close alignment between the topics discussed on Dr Math and the topics of the examination shows that an online tutorial system has utility because it reflects aspects of what is covered in the South African Mathematics curriculum. With the highest tutorial focus being on aspects considered challenging in the NSC reports, this also has the effect of independently validating the results of these reports.

Although the 2011–2013 NSC diagnostic reports were consulted, only the 2012 and 2013 NSC diagnostic reports contained the average performance figures for each question and its associated topic. As the topics listed were very general, a future study may analyse the question papers themselves to ascertain a greater list of topics, which would provide an even greater insight into the alignment with the topics discussed on Dr Math. This kind of information could be of value to the Dr Math tutors as it may allow them to present the learners proactively with questions on those topics, which they would not encounter in their daily classroom activities. In addition, it may address both the conceptual understanding of the learners and the concerns of the Department of Basic Education with regard to the amount of mathematical experience the learners gain outside of the classroom.

The authors declare that they have no financial or personal relationships which may have inappropriately influenced them in writing this article.