Acta Paedagogica Vilnensia ISSN 1392-5016 eISSN 1648-665X
2024, vol. 53, pp. 195–212 DOI: https://doi.org/10.15388/ActPaed.2024.53.13
Mohammad A. Tashtoush*
AL-Huson College University, AL-Balqa Applied University, Salt, Jordan
Faculty of Education & Arts, Sohar University, Sohar, Oman
Jadara University Research Centre, Jadara University, Jordan
https://orcid.org/0000-0002-2436-8155
E. mail: tashtoushzz@su.edu.om
Aida B. Qasimi
Faculty of Education and Arts, Sohar University, Sohar, Oman
E. mail: A_AlQasimi@su.edu.om
Nawal H. Shirawia
Faculty of Education and Arts, Sohar University, Sohar, Oman
E. mail: N_Shirawia@su.edu.om
Mamoon M. AL-Shannaq
Faculty of Educational Sciences, Sultan Qaboos University, Muscat, Oman
Faculty of Education, Yarmouk University, Irbid, Jordan
E. mail: mamoon.shanq@yu.edu.jo
Abstract. This study aimed to investigate the impact of an educational program rooted in the Program for International Student Assessment (PISA) on the mathematical achievement of tenth-grade female students. The study adopted a quasi-experimental approach, with two distinct groups: an experimental group exposed to the PISA-based educational program and a control group receiving conventional instruction. A meticulously crafted mathematical achievement test served as the primary assessment tool. The findings unequivocally revealed a substantial improvement in the mathematical achievement of both study groups post-test. Notably, significant differences in arithmetic means and standard deviations were observed between the groups, with the experimental group exhibiting markedly higher achievement scores. To determine the statistical significance of these differences, ANCOVA analysis was employed, alongside effect size calculations. In addition, the displayed results indicated statistically significant differences favoring the experimental group. In conclusion, this study underscores the substantial positive impact of the educational program based on the International Study for Student Assessment PISA on the mathematical achievement of tenth-grade female students. These results contribute to the body of knowledge on effective educational strategies and emphasize the potential benefits of integrating PISA-based educational programs into math instruction.
Keywords: Educational program; PISA; Mathematical achievement; Tenth-grade students, Assessment.
Santrauka. Tyrimo tikslas – ištirti Tarptautinio mokinių vertinimo programos (PISA) pagrindu parengtos ugdymo programos poveikį dešimtos klasės mergaičių matematikos pasiekimams. Tyrime taikytas kvazieksperimentinis metodas, jame dalyvavo dvi skirtingos grupės: eksperimentinė grupė, kuriai buvo taikoma PISA vertinimu grindžiama ugdymo programa, ir kontrolinė grupė, kuriai taikytas įprastinis mokymas. Pagrindinis vertinimo instrumentas buvo kruopščiai parengtas matematikos pasiekimų testas. Rezultatai atskleidė, kad abiejų tiriamųjų grupių matematikos testo rezultatai gerokai pagerėjo. Be to, tarp grupių nustatyti reikšmingi aritmetinių vidurkių ir standartinių nuokrypių skirtumai – eksperimentinės grupės pasiekimų įverčiai buvo gerokai aukštesni. Skirtumų statistiniam reikšmingumui nustatyti atlikta ANCOVA analizė ir apskaičiuotas efekto dydis, visi rezultatai atskleidė statistiškai reikšmingą eksperimentinės grupės pranašumą. Apibendrinant galima teigti, kad tyrimas parodė svarbų teigiamą tarptautinio PISA vertinimo pagrindu parengtos ugdymo programos poveikį dešimtos klasės mergaičių matematikos pasiekimams. Šie rezultatai papildo ugdymo proceso dalyvių žinias apie veiksmingas strategijas ir pabrėžia galimą PISA vertinimu grįstų ugdymo programų integravimo į matematikos mokymą naudą.
Pagrindiniai žodžiai: ugdymo programa, PISA, matematikos pasiekimai, dešimtos klasės mokinės, vertinimas.
_______
*Corresponding Author
Received: 14/08/2024. Accepted: 24/11/2024
Copyright © Mohammad A. Tashtoush
In recent years, there has been significant progress in the field of educational assessment, with education experts placing greater emphasis on it as part of educational reform and development efforts. Historically, educational reform primarily concentrated on inputs, but recent trends have shifted the focus towards educational outcomes, including the knowledge, skills, and attitudes acquired by students. This shift has given rise to international assessments, aimed at establishing a reliable database for comparing student performance against global benchmarks. These assessments facilitate cross-country comparisons based on the quality of education and educational outcomes, adhering to established international standards (Holliday, 2005). Reading proficiency is a primary target of international assessment tests, given that reading forms a significant portion of basic education curricula worldwide and plays a crucial role in early education stages (Wardat et al., 2023; Margaret & Clark, 2017).
Responding to these trends, several global initiatives and international projects have emerged to reform and enhance assessment processes, including the National Science Education Standards (NSES), Science, Technology, Society and Environment (STSE), Sciences, Technology, Engineering, and Mathematics (STEM), the Program of International Student Assessment (PISA), and the Next Generation Science Standards (NGSS). These initiatives aim to cultivate a creative generation capable of navigating rapid scientific and technological advancements (Burrows, 2018; Shirawia et al., 2024).
PISA stands out as one of the most significant research projects developed by the Organization for Economic Cooperation and Development (OECD). It serves as the principal international benchmark for assessing the efficiency and quality of educational systems across countries worldwide, with the goal of enhancing economic, social, and psychological well-being on a global scale (Karens & McNeely, 2009; OECD, 2019). Commencing in 2000 with the participation of 41 countries, PISA is conducted once every three years, yielding comparable data which enables nations to refine their educational policies and outcomes. Its results are increasingly influential in shaping national and international education policies. Today, PISA has become an indispensable tool for measuring and comparing national student performance, informing educational system enhancements, and contributing a wealth of information for evaluating the effectiveness of education systems and their alignment with international standards (Schleicher, 2017; Meng et al., 2017).
PISA has become instrumental in evaluating students’ competencies in knowledge, skills, and attitudes, aligning with the evolving educational curricula. PISA focuses on assessing students’ ability to apply their knowledge to various real-life scenarios, encompassing fundamental knowledge and skills in science, mathematics, and reading encountered in educational, domestic, and societal contexts (OECD, 2010). PISA assesses students’ proficiency in science, mathematics, and reading; the assessment prioritizes one of these fields, accounting for 70% of the test questions. For instance, the initial PISA test in 2000 concentrated on reading, followed by mathematics in 2003, science in 2006, and reading once more in 2009. Subsequent assessments revolved around mathematics in 2012, reading in 2015, and science in 2018 (Araujo et al., 2017). The eighth session of PISA, originally scheduled for 2021, was postponed due to the global pandemic, and was ultimately held in May 2022. It focused on mathematics and creative thinking.
PISA employs three formats: question booklets containing multiple-choice and essay questions in science, mathematics, and reading; questionnaires gathering information on students, their families, school principals’ perspectives on the school environment, and teacher profiles; and guides for coordinators, proctors, and test scorers. PISA stands out by not being tied to any specific curriculum but instead focuses on assessing students’ skills and knowledge in real-life situations that enable effective participation in society (OECD, 2004; Milena & Tsvetkova, 2016; Australian Council for Educational Research, 2010). PISA represents a collaborative effort among OECD member states, aiming to gauge students’ acquisition of fundamental skills and knowledge in science, mathematics, and reading (Yorulmaz et al., 2017; Bozkurt, 2014; Gurra, 2012; Urteaga, 2010; OECD, 2001).
In mathematics, PISA encompasses key areas, including mathematical knowledge, competencies, problem-solving contexts, student attitudes toward mathematics, and question difficulty levels. These competencies are intertwined and interrelated, involving mathematical thinking, modeling, proof, representation, communication, problem-solving, symbol and language use, mathematical operations, and evaluation of specific tasks, rather than compartmentalized mathematical knowledge (Bozkurt, 2014). The PISA assessment encompasses various mathematical dimensions, which are critical for evaluating students’ mathematical proficiency (OECD, 2010; 2004). These dimensions are divided into two main criteria: Mathematical Content Standards, and Mathematical Operations Standards.
The first dimension is the Mathematical Content Standard, emphasizing comprehension of mathematics by linking mathematical knowledge to individuals’ needs, enabling students to analyze, interpret, connect ideas, and reframe mathematical problems in diverse situations. This standard encompasses four areas: Space and Shapes: This dimension explores patterns and properties of shapes and situations encountered in various contexts, promoting spatial and geometric understanding. Change and Relationships: Natural phenomena often involve change, and this dimension assesses students’ ability to describe and model changing processes mathematically. It involves understanding relationships, rates of change, and the connections between different representations of change. Quantities: Quantitative literacy involves understanding relative sizes, recognizing numerical patterns, and using numbers to represent real-world quantities. Generalizing features in real-world scenarios is also a part of this dimension. Possibilities: This dimension deals with handling uncertain data. It includes data collection, analysis, presentation, interpretation, prediction, and drawing conclusions based on probability and statistics.
The second dimension Mathematical Operations Standards encompasses three areas: Formulating Mathematical Tasks: This criterion assesses students’ ability to identify variables, simplify hypotheses, and present mathematical structures and representations to comprehend or solve specific problems. Use of Mathematics: It involves the use of mathematical proofs, application of mathematical facts, procedures, and tools to obtain solutions, including work with algebraic expressions, data analysis, mathematical description development, and interpretation. Mathematics Interpretation: This dimension focuses on students’ capacity to critically assess mathematical solutions or results and interpret them in the context of the given problem. It involves evaluating mathematical solutions, reasoning within the problem’s context, and determining the logical and meaningful nature of the results.
According to (OECD, 2010; 2004), PISA categorizes mathematical questions into six levels, with each level representing a different depth of mathematical understanding: Level 1: At this level, students can answer questions related to familiar mathematical contexts with clear information and well-defined questions. They can identify information and follow direct instructions in straightforward situations. Level 2: Students at this level can interpret and distinguish situations in mathematical contexts which do not require more than direct deduction and extracting information. They can apply algorithms, formulas, or procedures and make direct inferences. Level 3: These students can execute procedures, including decision-making, and employ various strategies to solve problems. They can interpret and use examples from different information sources to draw direct inferences. Level 4: Students at this level can work effectively with complex, concrete mathematical situations that include constraints or assumptions. They can select and combine different examples, including symbolic ones, and relate them to real-world aspects. They demonstrate flexibility in using sophisticated skills and reasoning. Level 5: At this level, students can develop models for complex situations, work strategically with higher-order thinking skills, and evaluate different strategies for solving problems. They can criticize their actions and formulate interpretations and inferences. Level 6: These students can create models for complex problem situations, develop generalizations, and link various sources and examples of information, and transfer methods and strategies to tackle new situations. They display a profound understanding of mathematical concepts and can adapt to novel contexts effectively. Figure 1 illustrates the percentage of mathematical competencies attained by Jordanian students in PISA across all years, encompassing all six proficiency levels. (OECD, 2023).
Although Arab countries were not early participants in the international student assessment known as PISA, with only three Arab nations, namely, Jordan, Tunisia, and Qatar, taking part in the first three rounds of the test, their performance lagged behind in all three areas of evaluation. However, Jordan took a proactive stance by actively participating in the third session of the exam in 2006, and has consistently engaged in all the subsequent study sessions (2009, 2012, 2015, 2018, and 2022). The results of Jordanian students in the PISA tests have been incorporated into the performance indicators for the National Strategic Plan for Education in Jordan for 2025, as well as the quality of education indicators outlined in the Jordanian Ministry of Education’s strategy for the years 2018–2022, which places significant emphasis on the performance of Jordanian students in the PISA results (National Center for Human Resources Development, 2023; The Ministry of Education, 2018).
The outcomes of PISA reveal a noteworthy decline in students’ achievement levels, particularly in mathematics, across the study’s targeted fields. For instance, despite Jordan’s efforts to modernize its educational system and align it with global advancements, the country witnessed a significant drop in its students’ PISA results compared to international averages, signaling a cause for concern. This decline indicates a clear decrease in fundamental mathematical knowledge and skills among Jordanian students (Al-Ruwaished, 2020; National Center for Human Resources Development, 2023).
In the inaugural session of Jordanian student participation in 2006, the average performance in mathematics was 384 points, significantly below the international average of 494 points for all 57 participating countries, positioning Jordan at the 48th place globally. In 2009, Jordan topped the Arab world with a general average of 387 points, though it remained below the international average of 495 points for all 62 participating countries, while securing 50th place worldwide. In 2012, Jordan’s average performance reached 386 points, while the general average for the 65 participating countries was 494 points, earning Jordan 57th place internationally. In 2015, the general average dipped to 380 points, and, out of 72 participating countries, Jordan ranked 67th worldwide with an average of 490 points. Finally, in 2018, the general average improved to 400 points, though it still lagged behind the 459-point average for all 79 participating countries, resulting in a 65th-place international ranking. Figure 2 illustrates the performance of Jordanian students in PISA from 2006 to 2018, encompassing Jordan’s participation during this period. Finally, in the year of 2022, Jordan ranked fifth in the Arab world with the overall average of 395 points, and the overall average of 477 points for all participating countries (81), and a ranking of number 75 internationally; Figure 2 shows these results (The Ministry of Education, 2023).
Despite the significant challenges facing Jordan in the development and reform of its educational system, characterized by policies grounded in regional and global evidence and standards aligned with international assessments, and despite consistently subpar performance by Jordanian students in PISA tests over the years, falling below the required international benchmark, the results of previous iterations have shown a persistent decline in some areas and stability in others. This development was observed despite the exhaustive procedures, reforms, and preparations undertaken by the government to improve upon previous outcomes. An in-depth analysis of five consecutive rounds of participation by Jordanian students in these assessments has revealed a pronounced deficiency in mathematical skills and knowledge acquisition, a notable decline in academic attainment, and a clear inadequacy in students’ performance on PISA mathematics assessments.
In response to the recommendations stemming from the 2014 Educational Development Conference held in Jordan, which emphasized the continuous alignment of curricula with the findings of international assessment tests and the necessity of garnering insights from the educational field, a group of researchers, including mathematics teachers and supervisors, conducted field observations. These observations, carried out during various international and local tests, highlighted students’ shortcomings in fundamental skills and knowledge deemed essential, their underperformance in mathematics, and a general lack of comprehension regarding the significance, nature, and objectives of PISA.
To reinforce the urgency of implementing a program grounded in the PISA framework to elevate mathematical achievement, one can draw upon findings and recommendations from multiple prior studies (Yorulmaz et al., 2017; Araujo et al., 2017; Bozkurt, 2014; Gurra, 2012; Urteaga, 2010). Given the limited number of studies conducted within the Jordanian context utilizing the International Study for Student Assessment PISA as a foundation, this study addresses the primary question at hand: What is the impact of PISA-based educational program on the mathematical achievement?
The significance of this study lies in its endeavor to assess the impact of educational programs grounded in PISA on students’ mathematical achievements. It is anticipated that the findings of this research will provide valuable insights into elucidating the consistently modest performance of Jordanian students in PISA assessments over the five cycles, particularly in comparison to their peers from other participating countries. Such insights can pave the way for the formulation of strategies, initiatives, and remedial measures aimed at enhancing students’ performance in forthcoming test cycles. Moreover, the outcomes of this study can serve as a catalyst for the development of mathematics curricula with a specific focus on aligning them with the PISA international student assessment tests. This alignment can result in a more robust and comprehensive knowledge base, ultimately benefiting both educators and students.
This study has several key objectives. First and foremost, it seeks to construct an educational program rooted in the principles of the international assessment study PISA. This program’s primary aim is to get students acquainted with the nature and format of international assessment tests, ultimately leading to an improvement in their mathematical achievement. Furthermore, this research endeavors to shed light on the efficacy of a PISA-based training program in enhancing students’ mathematical performance. Through empirical investigation, it aims to demonstrate the program’s impact on raising the level of students’ achievement in mathematics. In addition to these objectives, the study aspires to formulate valuable recommendations and proposals. These suggestions are anticipated to play a pivotal role in the advancement of students’ academic achievements in mathematics across various academic levels.
• Sample Limitations: The research is confined to a specific group of participants, namely, tenth-grade female students who were actively engaged in their studies during the first semester of the 2022/2023 academic year. The results and conclusions drawn may not be applicable to other student populations or different grade levels.
• Temporal Constraints: The study was conducted within a specific time frame, focusing on the first semester of the 2022/2023 academic year. This temporal limitation may impact the generalizability of the findings to different time periods or academic terms.
• Spatial Boundaries: The research was carried out exclusively within the confines of the Yarmouk University Model School. Therefore, the results may not be representative of educational institutions with different characteristics, structures, or locations.
• Objective Constraints: The primary objective of this study is to assess the impact of a training program based on the international study for student assessment (PISA) on the development of mathematical achievement. Consequently, the study may not address other potential effects or outcomes related to educational programs.
• Methodological Constraints: The study’s conclusions are contingent on the tools and instruments employed, as well as their psychometric properties, specifically concerning validity and reliability. The research relies on these measures to achieve its intended scientific goals, but the limitations of these tools should be taken into account when interpreting the results.
• The Program of International Student Assessment (PISA): This is an international initiative involving participation from various countries worldwide. It evaluates the competencies of 15-year-old students in fundamental skills and knowledge encompassing the domains of science, mathematics, and reading. Furthermore, it facilitates a comparative analysis of student performances across the participating nations. This assessment is conducted on a triennial basis, starting from the year 2000.
• PISA-Based Educational Program: This refers to a meticulously crafted and organized collection of activities, exercises, procedures, strategies, methods, and instructional materials tailored in accordance with the International Student Assessment Study. These educational resources are integrated into the teaching process with the overarching goal of enhancing and elevating students’ mathematical proficiency.
• Mathematics Achievement: This term denotes the numerical score that students attain on the mathematical achievement examination meticulously designed by researchers to gauge their mathematical performance.
Upon conducting extensive research and investigations into previous studies concerning the International Student Assessment Study its correlations with various variables, it has become evident that there is a paucity of research in this area. Nonetheless, several studies have delved into interpreting and analyzing test results, as well as identifying crucial factors associated with enhancing student performance in PISA assessments. However, the current study seeks to use the PISA results as a tool for designing curricula rather than as a tool for evaluating academic achievement. This is perhaps the most distinctive feature of this study, making it one of the pioneering studies that utilize PISA results as a tool rather than as an evaluation, as highlighted by many previous studies. For instance, Schleircher (2009) conducted a study comparing the outcomes of various countries participating in PISA tests and explored factors contributing to improved student performance, notably, factors pertaining to students themselves, such as motivation, self-efficacy, utilization of the Internet, dictionaries, and training on questions akin to those in international tests like PISA.
The study of Anil and Duygu (2010) aimed to develop a structural equation model to examine the relationships between variables believed to influence the mathematics achievement scores of 15-year-old students in Turkey. Analysis of the established structural equation model revealed that the most significant predictor of students’ success in mathematics is time. The second most influential factor was education, indicating that higher parental education levels and an increased number of books at home positively impact students’ mathematics achievements. Additionally, the model demonstrated a significant positive relationship between success in mathematics and the environment. A positive but weak and inconsistent relationship was observed between the family (cultural wealth) and mathematics achievement.
Similarly, Gurra (2012) conducted a study aimed at identifying the most influential factors that bolster students’ performance in PISA tests in Mexico, Turkey, and Germany. The findings revealed that students with stronger economic backgrounds who had attended kindergarten exhibited greater confidence in their problem-solving abilities, resulting in higher PISA scores. Conversely, factors such as school absenteeism and punctuality issues had a negative impact on student performance.
In the context of Turkish students participating in the PISA study, Pektas & Kilinc (2016) investigated the connection between students’ self-efficacy and its influence on their academic achievements in mathematics within the framework of the international assessment study PISA. Their study also explored the relationship between mathematical achievement, educational effort, and the overall development of the educational system.
Zhang (2018) conducted research to ascertain the influence of behavioral control, problem-solving skills, and behavioral attributes on cognitive engagement during mathematics lessons among Chinese students. Examining educational expenditure, Harun & Hakan (2019) conducted a study to determine the relationship between the effectiveness of educational services provided to students and educational spending in Turkey. Despite substantial investments in education, there was a marked decline in the PISA test results. In 2019, Georghita (2019) investigated the impact of government spending on education in Romania on school performance, by utilizing PISA test results as a primary indicator. Despite high education expenditures, there was a decrease in the students’ PISA test scores.
In the Arab region, Alharbi (2020) explored the reasons behind low student performance in the Kingdom of Saudi Arabia in international assessment tests, particularly in mathematics, as perceived by male and female teachers and school principals. The study revealed pervasive low scores for Saudi Arabian students in PISA international mathematics assessments across various aspects. Moreover, there were no statistically significant differences in the reasons for low student performance based on variables like the current employment, gender, and qualifications.
In the Jordanian context, research on PISA has recently gained traction. Several studies have unveiled the primary reasons for low student performance in PISA tests, factors contributing to an improved performance in these tests, and the effectiveness of training programs rooted in the PISA framework. For instance, Yousef (2021) conducted a study demonstrating the efficacy of a PISA-based training program in enhancing tenth-grade students’ reading proficiency. Additionally, Al-Saeed (2022) conducted a study to identify the key reasons for low student performance in PISA mathematics tests from the perspectives of teachers and educational supervisors. These reasons encompassed factors associated with parents, teachers, students, and the curriculum.
Another study by Altwiessi and Alkasasbah (2022) explored the factors that contributed to the improved performance of Jordanian students in the 2018 PISA assessment. These factors encompassed various domains, including Ministry of Education policies, schools and their surrounding environment, teachers, students, and the curriculum. The study’s findings revealed that the respondents rated the overall influence of these factors on the student performance as high. Among the domains, factors related to the curriculum ranked first, followed by the Ministry of Education policies, teacher-related factors, student-related factors, and, finally, factors associated with schools and their surrounding environment.
The study of Wijaya and others (2024) adopted a qualitative approach, with 36 mathematics teachers in Indonesia selected randomly as its participants. Based on the teachers’ insights, the decline in mathematics scores in the 2022 PISA, compared to 2018 and 2015, is linked to six key factors: challenges related to the pandemic, curriculum issues, individual factors, limited resources, student-related factors, and parental involvement. In response, the teachers proposed several recommendations aimed at enhancing the student mathematics performance. These include the provision of additional training, upgrading ICT facilities in schools, and raising parental awareness about the significance of their support. The study’s findings present actionable suggestions for the government, schools, and mathematics teachers to improve the students’ mathematical performance and potentially boost PISA scores in 2025.
The present study employed a quasi-experimental design featuring two distinct groups: the experimental group, and the control group. In this research approach, an educational program was meticulously crafted, encompassing a range of activities, skills, and tasks designed to mirror the patterns found in PISA. This program was employed in instructing the students within the experimental group. A pre-and post-design was implemented to assess the two study groups. Specifically, the experimental group received instruction through the educational program, while the control group followed the conventional teaching methods.
The study was conducted with students from Yarmouk University Model School, focusing on tenth-grade female students during the first semester of the 2022/2023 academic year. A total of two sections were randomly selected for participation: the experimental group, comprising 45 female students who were taught by using an educational program based on PISA, and the control group, consisting of 48 female students who were taught by employing the traditional methods.
The mathematical achievement test was meticulously developed by drawing insights from existing research and theoretical literature (Yousef, 2021; Harun & Hakan, 2019; Zhang, 2018; Pektas & Kilinc, 2016). This test comprised 20 multiple-choice items structured to align with the levels of the PISA test questions. Additionally, a test blueprint was created to guide the test construction process. Each correct response was awarded one point, while incorrect answers received zero points. Consequently, scores on the test ranged from a minimum of 0 to a maximum of 20. The tool’s validity was established through expert review by a panel of arbitrators with substantial experience and expertise in the field. Based on their valuable feedback, necessary adjustments were made to ensure the tool’s appropriateness. Difficulty and discrimination coefficients were also calculated by administering the test to an exploratory sample of 20 female students from outside the study sample. Difficulty coefficients ranged from 0.49 to 0.75, while discrimination coefficients ranged from 0.36 to 0.71, reinforcing the tool’s validity for the study’s purposes. To assess the tool reliability, a test-retest approach was employed with a two-week interval, resulting in a Pearson correlation coefficient of 0.93. These values affirm the tool’s reliability for the study. The test administration time was determined by calculating the average duration required for all students in the exploratory sample, with the test taking approximately 75 minutes.
The educational program was developed after a comprehensive review of international PISA test models and guidelines for training mathematics teachers (National Center for Human Resources Development, 2013, 2017; OECD, 2019, 2010, 2004). This analysis informed the creation of educational content aimed at enhancing female students’ mathematical achievements. The program included a range of activities and exercises derived from the tenth-grade curriculum, aligned with PISA test questions, standards, foundations, and subject matter. Female students underwent training across three stages:
• Preparation: This stage introduced the students to the nature and significance of PISA tests, the program’s objectives, and the importance of participation. It also informed the students of Jordanian students’ historical performance in PISA tests, which had been consistently below international standards.
• Training: This phase exposed the students to the mathematical knowledge topics covered in the International Study for Student Assessment (PISA) tests and encouraged their engagement with a wealth of questions, activities, and exercises related to these topics. The educational material was taught over ten weeks, comprising 40 class sessions, each lasting 45 minutes. A table of specifications for the educational material was meticulously crafted to guide the teaching process.
• Evaluation: This stage aimed to assess the performance of trained female students to gauge the impact of the educational program on their achievement.
To ensure the program’s authenticity and relevance, it underwent scrutiny by a panel of arbitrators experienced in international tests. Their opinions and suggestions guided the program’s refinement to align with educational objectives and the requisite mathematical skills, knowledge, and competencies.
The study’s procedures were meticulously executed according to the following steps:
• Literature Review: Theoretical and research literature, alongside previous studies pertinent to the study’s subject, were thoroughly reviewed and integrated into the study tool and educational program preparation.
• Population Selection: The study population was determined by randomly selecting two groups: the experimental group, and the control group.
• Tool Development: The study tool was initially formulated, then submitted for arbitration, validation, and reliability assessment. Modifications were made based on the arbitrators’ recommendations.
• Educational Program Creation: An educational program based on the International Student Assessment Study (PISA) was developed, encompassing activities, exercises, and content aligned with the PISA questions, standards, and topics. Training occurred over three stages: preparation, training, and evaluation.
• Tool Validation: The tool was assessed by a committee of arbitrators experienced in international tests to ensure its suitability for achieving the study’s objectives.
The data was processed statistically by using the SPSS program. Descriptive statistics, including the mean scores and standard deviations, were computed to identify differences in the average scores. ANCOVA analysis was applied, and effect sizes were calculated to assess the impact of the educational program on the mathematical achievement test results.
The primary research question sought to uncover the impact of an educational program in the international study for student assessment on the mathematical achievement of tenth-grade female students. To address this question, the study computed the means and standard deviations for both the pre- and post-application scores of the mathematical achievement test. Table 1 provides a detailed overview of these findings.
|
Group |
Application |
Mean |
SD |
|
Control |
Pre-test |
٥.١٩ |
٢.٤٥٢ |
|
Post-test |
٦.٣٢ |
١.٩٥٦ |
|
|
Experimental |
Pre-test |
٥.٢١ |
٢.٤٥١ |
|
Post-test |
١٢.٨٢ |
١.٣٢٥ |
|
|
Max. score of the mathematical achievement test is 20 |
|||
Experimental and control groups. Notably, there are substantial disparities in the mean scores and standard deviations of the study sample members’ performance in the post-application of the mathematical achievement test. Specifically, the mean score for the experimental group stood at 12.82; this was in stark contrast to the control group’s mean score of 6.32. Moreover, the standard deviation for the experimental group amounted to 1.325, while the control group recorded a standard deviation of 1.956. To ascertain the statistical significance of these differences, a one-way analysis of variance (ANCOVA) was employed. Additionally, effect size calculations were carried out to evaluate the mathematical achievement in the post-application between the two study groups, with adjustments made to account for the influence of the pre-measurement (the associated variable). Table 2 presents the outcomes of these statistical analyses.
|
Source |
SS |
df |
MS |
F |
Sig. |
η2 |
Effect size |
|
Group |
١٨٦٩.٦٦ |
1 |
١٨٦٩.٦٦ |
٧٧.٥١ |
0.01٥* |
0.٥٩٨ |
Large |
|
Error |
٢١٩٥.١٩ |
٩١ |
٢٤.١٢ |
||||
|
Total |
٣٩٨٥.٩٦ |
٩٢ |
Table 2 unmistakably demonstrates statistically significant differences at a significance level of α = 0.05 between the arithmetic means of the two study groups concerning the achievement test. These differences are attributed to the impact of the educational program grounded in the international study for student assessment. Importantly, all of these differences favor the experimental group which received training through the educational program. To gauge the extent of the educational program’s effect on improving the achievement of the study participants, the effect size was determined by using eta square (η2), yielding a value of 0.598. This value signifies that the utilization of the educational program accounts for approximately 59.8% of the variance observed in the enhancement of the study sample’s achievement. The remaining unexplained variance 40.2% may be attributable to external influences or other factors.
The study’s results demonstrate that the participants in the experimental group achieved significantly higher scores on the mathematical achievement test, thereby signaling a substantial improvement in their academic performance. This notable progress can be attributed to their exposure to the educational program rooted in the international study for student assessment (PISA). Through this program, students probably had an opportunity to engage in a structured curriculum that introduced them to the standards and principles underpinning international assessments, providing them with a comprehensive understanding of these evaluations. Specifically, the program familiarized students with essential mathematical knowledge, competencies required for mastery, problem-solving contexts, and the complexity of questions typically encountered in the international study for student assessment (PISA) tests. It also included past question models tailored to the PISA format, the relevant mathematical content, and a range of engaging activities across various levels, which had previously been unfamiliar to the students. These activities were directly aligned with the criteria set forth by the international study for student assessment (PISA). Consequently, students actively engaged with the program’s content, leading to their remarkable performance improvement in mathematical achievement. This outcome is consistent with the findings of prior research studies (Yousef, 2021; Fannakhosrow et al., 2022; Tashtoush et al., 2022a), which demonstrated that participation in educational programs based on international assessments can substantially enhance students’ performance in achievement tests.
The incorporation of novel mathematical activities and mathematics-related content in the program infused an active and dynamic educational environment, fostering students’ enhanced comprehension of the nature of PISA international assessment tests. This encompassed a deeper grasp of the requisite mathematical knowledge, competencies, problem-solving contexts, and question levels encountered in PISA assessments. The program encouraged the use of mathematical symbols and operations, inspiring students to actively seek answers to the program’s questions and activities. As a result, the acquisition of mathematical skills became rooted in profound comprehension, nurturing a genuine desire for engagement and exploration. Learning in this manner cultivated a profound understanding of mathematical concepts.
The PISA-based educational program, which focuses on solving real-world problems, enhances students’ engagement with mathematics. This is because they perceive the importance of mathematics in their daily lives. When students understand how mathematics can be applied in real-life situations, it increases their interest in the subject and boosts their motivation to learn it. Additionally, it emphasizes critical thinking and problem-solving skills, which are skills that students prefer and perceive as practically valuable. These skills make students feel that learning mathematics is enjoyable and useful, thereby increasing their interest and motivation.
One of the goals of the PISA-based educational program is to make students more capable of addressing mathematical challenges in their practical lives. A heightened sense of self-efficacy further motivates students to engage with the subject, particularly when they succeed in completing challenging tasks. Students who undergo training programs based on PISA standards have demonstrated a significant increase in their desire to learn mathematics and a sense of accomplishment when solving practical mathematical problems compared to students in the traditional curricula.
In light of the aforementioned, and drawing from the insights of prior research (Schleircher, 2009; Pektas & Kilinc, 2016; Tashtoush et al., 2023b; Yousef, 2021), which consistently revealed performance enhancements, it is reasonable to attribute this improvement to the nature of the teaching activities and training embedded in the educational program founded on the international study PISA. This approach may provide motivation for learning, allowing every student to actively participate in educational content activities. Consequently, the program contributed to an improved academic performance and endowed students with a repertoire of skills applicable in various comparable situations. Familiarity with the nature and standards of international study PISA instilled a sense of interest and motivation for learning, thereby reinforcing the positive influence of the educational program. This, in turn, engendered a sense of confidence and assurance among the study participants when confronted with various international assessments, notably, the PISA tests.
In summary, the study’s findings lead to a significant conclusion: the educational program grounded in the international study PISA serves as a catalyst for building a strong foundation in mathematical knowledge. The results underscore that this program is an effective strategy for elevating academic achievement in mathematics, as evidenced by the significant advancements in the performance of female students. These findings align harmoniously with numerous prior studies (Tashtoush et al., 2020a; Rasheed et al., 2023; Tashtoush et al., 2020b; Tashtoush et al., 2023a). Ultimately, the study’s result showed that the improved performance of students in the mathematics achievement test within the experimental group can be attributed to the unique educational and learning experiences they encountered through the program. These experiences are challenging to replicate within a conventional learning environment and involve activities and content that promotes active student participation, while emphasizing the paramount importance of this instructional approach.
In light of the results of the study that support and promote the use of educational programs based on international studies, such as the Program for International Student Assessment PISA, to enhance students’ mathematical achievements, this study recommends the following:
• Expand the scope of interest among school administrations in international assessments, especially PISA, to foster a culture of awareness regarding the importance of international assessments and the necessity of participation in them.
• Provide training on educational programs like the one developed in this study based on the Program for International Student Assessment (PISA) to improve students’ mathematical achievements.
• Pay attention to the school curricula by focusing on activities and learning content that align with international assessments, especially the Program for International Student Assessment PISA.
• Conduct similar studies to the current study to explore the impact of the educational program developed based on the Program for International Student Assessment PISA on improving other variables, such as mathematical thinking, learning motivation, academic enthusiasm, and more.
References
Alharbi, M. (2020). The reasons for the decline of the students’ results in mathematics (PISA 2018) test in Saudia Arabia from the point of view of the test sample. Journal of Educational Sciences, 32(3), 589-618.
AL-Ruwaished, N. (2020). The effectiveness of teaching based on mathematical communication in developing the achievement of some concepts of coordinate geometry. Journal of Mathematics Education, 22(3), 311-333.
AL-Saeed, D. (2022). Reasons for Decline Performance of the (15) years old Students in the (PISA) in Mathematics from the viewpoint of teachers and Educational supervisors’ in Amman, Master Thesis, Middle East University, Jordan.
Altweissi, A, & Alkasasbah, A. (2022). Factors that contribute to the improvement of Jordanian students’ performance in the PISA 2018 from the perspective of the concerned stakeholders in Jordan. International Journal of research in Educational Sciences, 5(2), 391-450. http://iafh.net/index.php/IJRES/article/view/295
Anil, Duygu (2010). The analysis of factors affecting the mathematical success of Turkish students in the PISA 2006 evaluation program with structural equation modeling. American-Eurasian Journal of Scientific Research, 3(2), 222-227.
Araujo, L., Saltelli, A., & Schnepf, S. (2017). Do PISA data justify PISA-based education policy? International Journal of Comparative Education and Development, 19(1), 20-34. http://dx.doi.org.sdl.idm.oclc.org/10.1108/IJCED-12-2016-0023
Australian Council for Educational Research. (2010). PISA 2012 field trial problem solving framework. http://www.oecd.org/pisa/pisaproducts/46962005.pdf
Bozkurt, B. (2014). Development of reading literacy in south Korea from PISA 2000 to PISA 2009. Egitim Ve Bilim, 39(173), 140-154. https://search-proquest com. sdl.idm. oclc.org/docview/ 1521720027? accountid= 142908
Burrows, A., Lockwood, M., Borowczak, M., Janak, E., & Barber, B. (2018). Integrated STEM: Focus on informal education and community collaboration through engineering. Education Sciences, 8(4). https://doi.org/10.3390/educsci8010004
Fannakhosrow, M., Nourabadi, S., Huy, D., Trung, N., & Tashtoush, M. (2022). A comparative study of information and communication technology (ICT)-Based and conventional methods of instruction on learners’ academic enthusiasm for L2 learning. Education Research International. 2022, Article ID 5478088. https://doi.org/10.1155/2022/5478088
Gheorghita, D. (2019). Investment in education. Bulletin of the Transylvania University of Brasov, Serials VI: Medical Sciences, 12(2), 79-86.
Gurra, M. (2012). PISA 2012 Results in focus, what 15-year-olds know and what they can do with what they know. Retrieved from https://www.oecd.org/pisa/keyfindings/pisa-2012-results-overview.pdf
Hamidi, F., Soleymani, S., Dazy, S. & Meshkat, M. (2024). Teaching mathematics based on integrating reading strategies and working memory in elementary school. Athens Journal of Education, 11(1), 9-22. https://doi.org/10.30958/aje.11-1-1
Harun, K., & Hakan, Y. (2019). PISA Sonuçları ile Türkiye’de Eğitim Harcamaları İlişkisi. PISA Sonuçları ile Türkiye’de Eğitim Harcamaları İlişkisi. Bilgi, 21(2), 296-319.
Holliday, D. (2005). The effects of using blended learning strategy on academic achievement in a secondary social studies classroom. Abstract international, 57(1), 162-163.
Kamens, D., & McNeely, C. (2009). Globalization and the growth of international educational testing national assessment. Comparative Education Review, 54(1), 5-25. https://eric.ed.gov/?id=EJ873565
Küçükalioğlu, T. & Tuluk, G. (2021). The effect of mathematics teachers’ self-efficacy and leadership styles on students’ mathematical achievement and attitudes. Athens Journal of Education, 8(3), 221-238. https://doi.org/10.30958/aje.8-3-1
Margaret, M., & Clark, O. (2017). Scotland’s developing literacy policies: one response to PISA, Education Journal, 11(3), 24-26.
Meng, L., Muoz, M., Hess, K. (2017). Effective teaching factors and student reading strategies as predictors of student achievement in PISA 2009: The case of China and the United States, Educational Review, 69(1), 68–84. http://www.tandf.co.uk/journals
Milena I., & Tsvetkova, A. (2016). The shadows of reading: Reasons for the bad results of Bulgarians in PISA studies. European Journal of Contemporary Education, 17(3), 368-377. https://doi.org/10.13187/ejced.2016.17.368
National Center for Human Resources Development. (2013). A guide for Arabic teachers to deal with students’ learning errors in the light of their findings on PISA questions. Jordan. Retrieved from: http://www.moe.gov.jo/Files/(15-2-2018)(3-38- 22%20PM).pdf
National Center for Human Resources Development. (2017). Program for International Student Assessment, National Report. Amman: The Center’s Publications Series (184).
National Center for Human Resources Development. (2023). Program for International Student Assessment, National Report. Amman: The Center’s Publications Series.
OECD. (2001). Measuring Student Knowledge and Skills: A New Framework for Assessment.
OECD. (2004). PISA 2003 Assessment Framework: Mathematics, Reading, Science and Problem-Solving Knowledge and Skills.
OECD. (2010). PISA 2009 Assessment Framework - Key competencies in reading, mathematics and sciences.
OECD. (2023). PISA 2018 Results (Volume I): What students know and can do, PISA. OECD Publishing. https://doi.org/10.1787/5f07c754-en
Pektas, S., & Kilinc, M. (2016). The relationship between different variable for math literacy in PISA 2012: A hybrid model study. Mehmet Akif Ersoy Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 8(17), 34-50. https://doi.org/10.20875/sb.41469
Rasheed, N. & Tashtoush, M. (2021). The Fertility and its Relation with some demographic, economic and social variables in Jordan. Turkish Journal of Computer and Mathematics Education, 12(11), 5088-5095.
Rasheed, N., & Tashtoush, M. (2023). The impact of cognitive training program for children (CTPC) to development the mathematical conceptual and achievement. Journal of Higher Education Theory and Practice, 23(10), 218-234.
Schleicher, A. (2017). How your school compares internationally OECD test for schools (Based on PISA). Sample report high school OECD demo district. https://www.oecd.org/pisa/aboutpisa/Golden_e-book_1_example.pdf
Schleircher, A. (2009). Securing quality and equity in education: Lessons from PISA. UNESCO Publication-Cairo, 39(3), 251-263.
Shirawia, N., Qasimi, A., Tashtoush, M., Rasheed, N., Khasawneh, M., & Az-Zo’bi, E. (2024). performance assessment of the calculus students by using scoring rubrics in composition and inverse function. Applied Mathematics and Information Sciences, 18(5), 1037-1049.
Tashtoush, M., Alali, R., Wardat, Y., AL-Shraifin, N., & Toubat, H. (2023 b). The impact of information and communication technologies (ICT)-based education on the mathematics academic enthusiasm. Journal of Educational and Social Research, 13(3), 284-293, Poland.
Tashtoush, M., Alshunaq, M., & Albarakat, A. (2020 a). The Effectiveness of self-regulated learning (SRL) in creative thinking for CALCULUS students. PalArch’s Journal of Archaeology of Egypt/ Egyptology. 17(7), 6630-6652. https://archives.palarch.nl/index.php/jae/article/view/3022
Tashtoush, M., Shannaq, M., & Barakat, A. (2020 b). The effect of using self-regulated learning strategy to reduce the level of mathematics anxiety among students of Al-Huson University College. Jordanian Journal of Education, Jordanian Association for Educational Sciences, 5(3), 306-329.
Tashtoush, M., Wardat, Y., Aloufi, F., & Taani, O. (2022 a). the effect of a training program based on TIMSS to developing the levels of habits of mind and mathematical reasoning skills among pre-service mathematics teachers. Eurasia Journal of Mathematics, Science and Technology Education, 18(11), em2182. https://doi.org/10.29333/ejmste/12557
Tashtoush, M., Wardat, Y., Aloufi, F., & Taani, O. (2022 b). The effectiveness of teaching method based on the components of concept-rich teaching in achievement for students of linear algebra course and their attitudes towards. Journal of Higher Education Theory and Practice, 22(7), 41-57. https://doi.org/10.33423/jhetp.v22i7.5269
Tashtoush, M., Wardat, Y., & Elsayed, A. (2023 a). Mathematics distance learning and learning loss during COVID-19 pandemic: Teachers’ perspectives. Journal of Higher Education Theory and Practice, 23(5), 162-174.
Teaga, E. (2010). Los resultados del estudio PISA en francia results of the PISA survey in france. Revista Complutense De Education, 21(2), 231-244. https://revistas.ucm.es/index.php/RCED/article/view/RCED1010220231A
The Ministry of Education. (2023). PISA international test. Amman: Jourdan. http://www.moe.gov.jo/en/node/23527
Wardat, Y., Tashtoush, M., Alali, R., & Jarrah, A. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. EURASIA Journal of Mathematics, Science and Technology Education, 19(7), 1-18, Article No: em2286.
Wijaya, T., Hidayat, W., Hermita, N., Alim, J., & Talib, C. (2024). Exploring contributing factors to PISA 2022 mathematics achievement: Insights from Indonesian teachers. Infinity, 13(1), 139-156.
Yorulmaz, Y., Çolak, C., & Ekinci, A. (2017), An Evaluation of PISA 2015 achievements of OECD countries within income distribution and education expenditures. Turkish Journal of Education, 6(4), 169-185. DOI:10.19128/turje.329755
Yousef, A. (2021). The effect of training on tests forms of program for international student assessment (PISA) in improving tenth grade students’ performance in reading comprehension tests in Irbid directorate. AL-Najah University Journal for Social Sciences, 35(2), 199-220. https://journals.najah.edu/article/1863/
Zhang, H. (2018). Individual cognitive and contextual factors affecting Chinese students’ mathematical literacy: a hierarchical linear modeling approach using program for international student assessment PISA 2012 [Doctoral dissertation, Kent State University]. Ohio LINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=kent1531401210902013