Español

Evaluation 3.3

Does Tutoring Improve Learning?

Tem+Matemática: Improving Educational Performance in Brazil

Students often complain that math is difficult to learn, and for many, it is far from their favorite subject. Improving performance is a considerable challenge because the subject is often not taught in an interesting way and not related to problems relevant to students.

The state of São Paulo, Brazil launched the Tem+Matemática initiative to motivate students to improve their math skills, which are considered critical in order to access more complex knowledge and function in society.

Background

Since the 1990s, there has been a gradual but significant increase in Brazil in the enrollment rates of school-age youth. This development has substantially changed the student profile and brought about major challenges for public education systems. The students who enrolled in the education system increasingly come from less-privileged socioeconomic groups and aspire to attain an education that surpasses that of their parents.

However, this increase in enrollment has not been accompanied by improvements in the quality of instruction. The higher enrollment rates often mask important issues: high repetition rates, low completion rates of educational cycles, and poor performance on international tests such as the Programme for International Student Assessment (PISA). For example, on the 2012 PISA tests, 67 percent of 15-year-old Brazilian students did not attain minimum learning levels to carry out simple algebraic equations and/or calculate proportions or areas. That rate is three times higher than the rate for countries in the Organisation for Economic Cooperation and Development (OECD).

Intervention

With this situation in mind, the Foundation Institute for Economic Research (Fundación Instituto de Pesquisas Económicas), the Education Secretariat of the State of São Paulo, and the IDB designed Tem+Matemática. The program provided tutoring for students in the seventh and ninth grades who had problems learning math and had a desire to overcome those problems. Study (tutoring) groups made up of three to five students from the same grade and a tutor were formed and met periodically until the end of the school term.

The tutors were university students in mathematics or other areas such as chemistry and physics. By working in an informal environment, students could show the tutor what specifically was giving them problems or could review work from the class, with guidance of the tutor. The connection with the tutors – who often came from the same socioeconomic background as the students and thus demonstrated that the challenges involved in attaining a university degree could be overcome – was considered key for the program to be a success.

Study sessions lasted 90 minutes and were held between August and November of 2011. The time devoted to these sessions, on top of class time, amounted to 40 percent of the class time assigned for math during the school year. The objective was for the tutoring sessions to bring about an improvement in student learning.

Evaluation

To determine the effectiveness of the program, an experimental impact evaluation supported by the IDB examined four dimensions: performance in mathematics, Portuguese, science, and geography; dropout, retention, and absenteeism rates; non-cognitive skills such as perseverance, self-esteem, self-efficacy, socialization (willingness to work in group settings and networks of friends, among others), and autonomy; and study habits (strategies for study and time devoted to mathematics).

The Education Secretariat selected 1,200 schools with the lowest average math scores. Among these, 210 schools were selected that had seventh and ninth grade classes and that were located near universities with bachelor’s degree programs. Students were then given the opportunity to participate in the tutoring sessions. Finally, 142 schools were selected that had the physical space available for the sessions and where at least 10 students signed up for the tutoring and three tutors expressed interest in participating. The schools were divided into an 88 school treatment group (those that participated in the tutoring program) and a control group comprised of 54 schools. The evaluation was conducted with the pool of students who participated in at least one-third of the tutoring sessions.

The evaluation estimated the short-term effects upon completion of the program and the long-term effects two or three years after the program had ended. The results showed: little short-term impact in terms of improved learning; positive effects on indicators of perseverance and willingness to work in a group setting; and negative effects, although not statistically significant, on retention, dropout, and absenteeism rates. No evidence was found of long-term effects on the variables evaluated.

Modest Results, Useful Lessons

Since it had no impact on any of the variables of interest, why is it worth reviewing the experience of this program? The answer is because it provides lessons for future endeavors. Issues that came up during program implementation included the following:

  • Problems with recruiting interested tutors resulted in frequent absenteeism of both students and tutors, mainly during the initial weeks of the project. What made recruitment and retention of tutors most difficult was the distance between the schools where the tutoring took place and the universities where the tutors were studying, as well as the low and unattractive remuneration for the tutors during a time when they had other employment possibilities in the labor market.
  • Students frequently dropped out of the tutoring sessions, many of them because of the absence of the tutors during the first week of the program. Nearly 65 percent of the students who originally enrolled or were offered an opportunity to participate in the program never attended a session, or attended less than 10 of the 30 planned sessions.

Without a doubt, there is a need for increased availability and better coordination of this type of programs. In terms of the tutors, it is important to strengthen their understanding of teaching practices before they graduate. In terms of the students, their teachers should take on a more active role in recommending them for the tutoring sessions and following up on their actual participation. Finally, it is essential that coordination between math teachers and tutors be improved with regard to the activities to be covered in the tutoring sessions and to overcome the challenges in learning mathematics and also with teacher training institutions in order to support the tutors along the programs.

Story by:
Marcelo Pérez Alfaro, a lead education specialist at the IDB in Brazil.

More about the evaluation:
Fundação Instituto de Pesquisas Econômicas. (2015). “Avaliação do Impacto de Programa Tem Mais Matemática,” Relatorio Final de Resultados.

Continue reading Chapter 3 - Evaluating Projects to Enhance Learning and Policy Making