PISA and Beyond: Analysis of International Data from PISA Using Multiple Tools and Incorporating Additional Country-Level Data

There has been growing interest in examining the performance of students cross-nationally on international assessments and trying to better understand the characteristics and initiatives of the top performing education systems (National Governors Association, CCSSO, & Achieve, 2008; OECD, 2010; Fordham Institute, 2009). PISA, a study by the Organization for Economic Cooperation and Development (OECD), contributes to this work. Begun in 2000 and conducted every three years, it /includes an assessment portion as well as questionnaires administered to 15-year-old students and their principals to obtain contextual information at the student and school levels.

Given that that the aim of PISA is to “assess the cumulative yield of education systems at an age where compulsory schooling is still largely universal” (OECD, 2009, p. 11) and acknowledging that the test assesses learning both in school and outside of school, the information obtained from PISA will benefit by incorporating country-level information from outside sources. The Country Profiles Tool (http://198.106.21.94/DefaultCountry.aspx) and the International Cross-Time, Cross-System Education Database (XTXS) (http://www.intledstatsdatabase.org/) include data from outside sources such as the World Bank and UNESCO. Using these tools, for example, researchers could examine how findings may differ for the OECD countries compared to the PISA-participating non-OECD countries (which include many less developed countries). Consistent with the CIES 2014 conference theme, “Revisioning Education For All,” PISA along with these tools inform research and policy discussions regarding educational equality, inclusion, equity of learning outcomes, and innovation both internationally and within nations.

This workshop will benefit researchers with varying levels of statistical training by presenting two major sets of data tools:

First, we will present tools for analyzing PISA data. These include those that require commercial statistical software—Stata, SAS macros, and the IEA IDB Analyzer (http://www.iea.nl/data.html), an add-on to SPSS. A free alternative is AM Statistical Software (http://am.air.org). We will also demonstrate a free easy-to-use web tool, the PISA International Data Explorer (PISA IDE) (http://nces.ed.gov/surveys/international/ide/), whereby users can run statistical tests and create customized tables, figures, and comparative maps to explore student performance across countries, subjects, and contextual factors.

Second, we will present tools that go beyond PISA in answering research questions—showing participants how to (a) use country-level data from other sources in conjunction with PISA results and (b) how to merge country-level variables to PISA data files. For the former, the Country Profiles Tool can be used to compare findings for more than 90 countries across several core indicators: international assessments, education, demographics, government, and economics. For the latter, the XTXS database /includes country-level education-related data for 223 countries and as far back as 1975. Some of the major topic areas include student outcomes, education system characteristics, economic characteristics, and population and health characteristics.

Target Audience

The target audience is CIES members—academics, students, and practitioners—interested in doing comparative international research. Some portions of the workshop will be directed to researchers with experience using one of the common statistical software packages (SAS, SPSS, or Stata), although considerable time will be spent demonstrating web tools that don’t require statistical training.

Instructional Goals

We expect to facilitate a half-day workshop for up to 75 people. This workshop will teach data users how to access resources and analyze assessment and contextual data from PISA. Multiple tools to assist people with varying research skills and interests will be presented, including many that are free and publicly available. Participants will be shown how PISA data can be used in conjunction with or linked to additional country-level data. The workshop will focus on newly released data from PISA 2012, in which 65 countries/economies participated. Unique aspects of PISA 2012 are the inclusion of computer-based assessments in mathematics literacy, reading literacy, and general problem solving, and an assessment of students' financial literacy.

Planned Workshop Activities

This workshop will include presentations on PISA and the analysis of PISA data, focusing on the unique aspects of the most recent PISA 2012 administration. There will be demonstrations of how research questions can be addressed using various analytic tools. There will be time for participants to explore their own research questions, with time allocated at the end of each presentation for participants to work with the data and data tools while instructors answer questions and provide hands-on assistance. Thus, participants will be able to do comparative and international education research using multiple analytic tools with data at multiple levels (i.e., student, school, and country levels) from multiple sources. It is recommended that participants bring a laptop to follow along with the demonstrations and participate in hands-on exercises.

Instructional Staff

David C. Miller, Ph.D., American Institutes for Research
Andrea Arce-Trigatti, Graduate Student, Vanderbilt University, Peabody College
Ngaire Honey, Graduate Student, Vanderbilt University, Peabody College
Bao Le, Graduate Student, Carnegie Mellon University, Heinz College
Christina Scheller, American Institutes for Research