Unveiling PISA: A Comprehensive Guide to the International Student Assessment

The Programme for International Student Assessment (PISA) stands as a significant worldwide study conducted by the Organisation for Economic Co-operation and Development (OECD). It involves both member and non-member nations and aims to evaluate educational systems by measuring the scholastic performance of 15-year-old pupils in mathematics, science, and reading. First performed in 2000, it is repeated every three years, with the exception of a one-year delay in the current cycle (from 2021 to 2022) due to the pandemic. After the 2025 data collection, PISA will change to a 4-year data collection cycle. PISA's primary goal is to provide comparable data, enabling countries to improve their education policies and outcomes. This article delves into the intricacies of PISA, its methodologies, impacts, and the debates surrounding its use and interpretation.

Historical Context and Evolution

Prior to the 1990s, national tests were not widely used in many European countries. However, the landscape began to shift in the 1990s, with ten countries or regions introducing standardized assessments. Since the early 2000s, an additional ten have followed suit. PISA stands in a tradition of international school studies, undertaken since the late 1950s by the International Association for the Evaluation of Educational Achievement (IEA). PISA 2022 results represent outcomes from the 8th cycle of PISA since its inception in 2000.

The Aims and Objectives of PISA

PISA emphasizes functional skills that students have acquired as they near the end of compulsory schooling. The assessment focuses on evaluating how well students can apply their knowledge to real-world situations, rather than simply testing their recall of facts. Its aim is to provide comparable data to enable countries to improve their education policies and outcomes.

Assessment Domains and Focus

PISA assesses students in three core domains:

Reading Literacy: Evaluating students' ability to understand, use, reflect on, and engage with written texts to achieve their goals, develop their knowledge and potential, and participate in society.
Mathematics Literacy: Assessing students' capacity to formulate, employ, and interpret mathematics in a variety of contexts, including reasoning mathematically and using mathematical concepts, procedures, facts, and tools to describe, explain, and predict phenomena. The PISA mathematics literacy test asks students to apply their mathematical knowledge to solve problems set in real-world contexts. To solve the problems students must activate a number of mathematical competencies as well as a broad range of mathematical content knowledge.
Science Literacy: Measuring students' ability to engage with science-related issues, and with the ideas of science, as a reflective citizen.

Each cycle of PISA emphasizes one subject (math, reading, or science) at a time, with literacy skills in the spotlight for the most recent iteration. PISA also assesses students in innovative domains. In 2012 and 2015 in addition to reading, mathematics and science, they were tested in collaborative problem solving.

Read also: PIC: Your Path to Higher Education

Target Population and Sampling Methodology

The students tested by PISA are aged between 15 years and 3 months and 16 years and 2 months at the beginning of the assessment period. The school year pupils are in is not taken into consideration. Only students at school are tested, not home-schoolers. In PISA 2006, however, several countries also used a grade-based sample of students. To fulfill OECD requirements, each country must draw a sample of at least 5,000 students. In small countries like Iceland and Luxembourg, where there are fewer than 5,000 students per year, an entire age cohort is tested. Notably, despite being randomly selected from pre-approved schools in participating countries, students who take the PISA must be demographically representative of their country or region.

Test Format and Structure

Each student takes a two-hour computer based test. Part of the test is multiple-choice and part involves fuller answers. There are six and a half hours of assessment material, but each student is not tested on all the parts. Following the cognitive test, participating students spend nearly one more hour answering a questionnaire on their background including learning habits, motivation, and family. School directors fill in a questionnaire describing school demographics, funding, etc. In 2012 the participants were, for the first time in the history of large-scale testing and assessments, offered a new type of problem, i.e.

Data Analysis and Scaling

From the beginning, PISA has been designed with one particular method of data analysis in mind. Since students work on different test booklets, raw scores must be 'scaled' to allow meaningful comparisons. This generation of proficiency estimates is done using a latent regression extension of the Rasch model, a model of item response theory (IRT), also known as conditioning model or population model. The proficiency estimates are provided in the form of so-called plausible values, which allow unbiased estimates of differences between groups. The latent regression, together with the use of a Gaussian prior probability distribution of student competencies allows estimation of the proficiency distributions of groups of participating students. The scaling and conditioning procedures are described in nearly identical terms in the Technical Reports of PISA 2000, 2003, 2006.

Presentation and Interpretation of Results

All PISA results are tabulated by country; recent PISA cycles have separate provincial or regional results for some countries. Most public attention concentrates on just one outcome: the mean scores of countries and their rankings of countries against one another. In the official reports, however, country-by-country rankings are given not as simple league tables but as cross tables indicating for each pair of countries whether or not mean score differences are statistically significant (unlikely to be due to random fluctuations in student sampling or in item functioning). PISA never combines mathematics, science and reading domain scores into an overall score. However, commentators have sometimes combined test results from all three domains into an overall country ranking. Spiegelhalter of Cambridge wrote: "Pisa does present the uncertainty in the scores and ranks - for example the United Kingdom rank in the 65 countries is said to be between 23 and 31.

PISA's Influence on National Education Policies

Emerging research suggests that international standardized assessments are having an impact on national assessment policy and practice. PISA may influence national education policy choices in a variety of ways. Against this, impact on national education systems varies markedly. National policy actors refer to high-performing PISA countries to "help legitimise and justify their intended reform agenda within contested national policy debates". In such instances, PISA assessment data are used selectively: in public discourse governments often only use superficial features of PISA surveys such as country rankings and not the more detailed analyses. Recent decades have witnessed an expansion in the uses of PISA and similar assessments, from assessing students' learning, to connecting "the educational realm (their traditional remit) with the political realm".

Read also: USA Degree Programs

Country-Specific Performance and Reactions

Several countries have shown notable trends in their PISA performance, eliciting diverse reactions and policy adjustments.

China: China's participation in the 2012 test was limited to Shanghai, Hong Kong, and Macau as separate entities. In 2012, Shanghai participated for the second time, again topping the rankings in all three subjects, as well as improving scores in the subjects compared to the 2009 tests. Shanghai's score of 613 in mathematics was 113 points above the average score, putting the performance of Shanghai pupils about 3 school years ahead of pupils in average countries. Andreas Schleicher, PISA division head and co-ordinator, stated that PISA tests administered in rural China have produced some results approaching the OECD average. Critics of PISA counter that in Shanghai and other Chinese cities, most children of migrant workers can only attend city schools up to the ninth grade, and must return to their parents' hometowns for high school due to hukou restrictions, thus skewing the composition of the city's high school students in favor of wealthier local families.
United Kingdom: In the 2012 test, as in 2009, the result was slightly above average for the United Kingdom, with the science ranking being highest (20). England, Wales, Scotland and Northern Ireland also participated as separated entities, showing the worst result for Wales which in mathematics was 43rd of the 65 countries and economies. The United Kingdom had a greater gap between high- and low-scoring students than the average. There was little difference between public and private schools when adjusted for socio-economic background of students.
Finland: Finland, which received several top positions in the first tests, fell in all three subjects in 2012, but remained the best performing country overall in Europe, achieving their best result in science with 545 points (5th) and worst in mathematics with 519 (12th) in which the country was outperformed by four other European countries. For the first time Finnish girls outperformed boys in mathematics narrowly. It was also the first time pupils in Finnish-speaking schools did not perform better than pupils in Swedish-speaking schools.
India: India participated in the 2009 round of testing but pulled out of the 2012 PISA testing, with the Indian government attributing its action to the unfairness of PISA testing to Indian students. India had ranked 72nd out of 73 countries tested in 2009. Accordingly, in February 2017, the Ministry of Human Resource Development under Prakash Javadekar decided to end the boycott and participate in PISA from 2020. To address the socio-cultural disconnect between the test questions and students, it was reported that the OECD will update some questions.
Sweden: Sweden's result dropped in all three subjects in the 2012 test, which was a continuation of a trend from 2006 and 2009. It saw the sharpest fall in mathematics performance with a drop in score from 509 in 2003 to 478 in 2012.

Criticisms and Controversies

PISA has faced criticism from various academics and educational experts, raising concerns about its methodology, interpretation, and impact.

Statistical Validity: Queen's University Belfast mathematician Dr. Hugh Morrison stated that he found the statistical model underlying PISA to contain a fundamental, insoluble mathematical error that renders Pisa rankings "valueless". Goldstein remarked that Dr. Morrison's objection highlights "an important technical issue" if not a "profound conceptual error".
Overstatement of Capabilities: Goldstein cautioned that PISA has been "used inappropriately", contending that some of the blame for this "lies with PISA itself. I think it tends to say too much for what it can do and it tends not to publicise the negative or the weaker aspects."
Lack of Transparency: Professors Morrison and Goldstein expressed dismay at the OECD's response to criticism. Morrison said that when he first published his criticisms of PISA in 2004 and also personally queried several of the OECD's "senior people" about them, his points were met with "absolute silence" and have yet to be addressed. Professor Svend Kreiner, of the University of Copenhagen, agreed: "One of the problems that everybody has with PISA is that they don't want to discuss things with people criticising or asking questions concerning the results. They didn't want to talk to me at all.
Socio-cultural Disconnect: The Indian government attributed its decision to pull out of PISA testing to the unfairness of PISA testing to Indian students, citing a socio-cultural disconnect between the questions and Indian students.

Alternative Assessments and Comparisons

While PISA is a prominent international assessment, it is not the only one. The Trends in International Mathematics and Science Study (TIMSS) is another widely recognized assessment. TIMSS, on the other hand, measures more traditional classroom content such as an understanding of fractions and decimals and the relationship between them (curriculum attainment).

PISA and Proficiency Levels

PISA reports student proficiency in terms of levels of proficiency. PISA items are mapped on a single continuum. The difficulty of items is estimated by considering the percentage of students getting each item correct. Each level is defined by the items that students at that level are likely to get correct. Table A-2 presents a summary description of the competencies associated with each of the proficiency levels on the PISA reading literacy scale.

Contemporary Challenges and Future Directions

According to a 2023 book, PISA is failing in its mission. Kijima and Lipscy note that the databases generated by large-scale international assessments have made it possible to carry out inventories and comparisons of education systems on an unprecedented scale on themes ranging from the conditions for learning mathematics and reading, to institutional autonomy and admissions policies. They allow typologies to be developed that can be used for comparative statistical analyses of education performance indicators, thereby identifying the consequences of different policy choices.

Read also: Undergraduate Admissions at Oxford

tags: #international #student #assessment #pisa #explained