Which problem-solving research-based assessment should I use in my class?
This recommendation initially appeared as an article in the American Journal of Physics: A. Madsen, S. B. McKagan, E. C. Sayre, and C. A. Paul, Resource Letter RBAI-2: Research-based assessment instruments: Beyond physics topics. Am. J. Phys, 87, 350 (2019).
Students’ ability to solve a problem when there is no solution method obvious to the solver (Mayer 1992) is a key skill that many physics faculty would like their students to develop. Problem-solving can be defined in many ways, e.g., the ability to solve physics textbook problems (Marx and Cummings 2010) or a collection of many components that a solver brings to bear to solve any problem, regardless of discipline (Adams and Wieman 2015). Because of the variety of interpretations of what problem solving means, there are also a variety of instruments to measure different aspects of problem solving, including the Minnesota Assessment of Problem Solving rubric (MAPS) (Docktor et al. 2016), the Colorado Assessment of Problem Solving (CAPS) (Adams and Wieman 2015), and the Assessment of Textbook Problem Solving Ability (ATPSA) (Marx and Cummings 2010). There are also surveys to probe students attitudes about problem-solving, rather than their skills (AAPS and APSS). These are discussed in the "Beliefs about physics learning in a specific context" section in this expert recommendation. To learn more about the research in problem solving, primarily in physics, see “Resource Letter RPS-1: Research in problem solving.”
Table1. Problem-solvingassessments.
| Title | Focus | Intended Population | Format | Research Validation | Purpose |
|
Rubric to score written problem solutions |
High school, intro college |
Rubric |
Silver |
Assess written problem solutions on five different aspects of problem solving in undergraduate introductory physics courses. |
|
|
Detailed understanding of students’ problem solving |
Graduate, upper-level, intermediate, intro college, high school, middle school |
Open-ended |
Silver |
Assess students’ strengths and weaknesses on 44 different components of the problem-solving process, using a general problem-solving situation that is not tied to any specific discipline. |
|
|
Solving textbook problems |
Intro college |
Open-ended |
Bronze |
Gauge students’ problem-solving ability in a first-semester calculus-based physics course. |
Minnesota Assessment of Problem Solving (MAPS)
The Minnesota Assessment of Problem Solving rubric (MAPS) (Docktor et al. 2016) is a rubric that you can use to score your students’ written solutions using the following 5 categories of problem-solving: (1) useful description, (2) physics approach, (3) specific application of physics, (4) mathematical procedures, and (5) logical progression. The MAPS rubric is applicable to a wide variety of problem types and introductory physics topics. With this rubric, you score each student’s written solution from 1 to 5 for each category, and then, look at the frequency of rubric scores for each category across the students in your class to get a sense of their problem-solving strengths and weaknesses. The MAPS rubric has been used at the high school and introductory college level. This rubric was created based on years of research on student problem solving at the University of Minnesota (Heller et al. 1992, Blue 1997, Foster 2000) and has been extensively studied for evidence for validity, reliability, and utility (Docktor 2009).
Colorado Assessment of Problem-Solving (CAPS)
The Colorado Assessment of Problem Solving (CAPS) (Adams and Wieman 2015) is an open-ended problem-solving assessment which presents a general problem situation from the Jasper Woodbury Series that is not tied to any specific discipline, so that students do not have to understand any particular physics concept in order to complete the assessment. The CAPS consists of a script describing a scenario and questions about how to solve the problems in that scenario. Students’ responses to the questions are graded on a continuum using a rubric that assesses 44 different sub-skills of the problem-solving process, to gauge students’ strengths and weaknesses in problem solving. There is no overall score, as the CAPS is meant to help you assess which aspects of problem solving an individual student needs more help with. It is appropriate for any level of student (middle school to graduate students). These 44 sub-skills are divided into three categories as follows: (1) knowledge; (2) beliefs, expectations, and motivation; and (3) processes. Use it to give individual guidance to specific students, e.g., undergraduate research student and graduate student. It would not be appropriate to use to assess problem solving as a whole in your class.
Assessment of Textbook Problem Solving Ability (ATPSA)
The Assessment of Textbook Problem Solving Ability (ATPSA) (Marx and Cummings 2010) contains open-ended problems similar to the end of chapter textbook problems. The content covered on the ATPSA is intentionally limited to Newton’s laws, energy, and momentum, as these are commonly taught topics in introductory courses. The ATPSA is meant for introductory undergraduate calculus-based mechanics courses, uses right/ wrong grading, and can be given as a pre- and post-test, so the overall results can be used to evaluate a course (but not individual students). The ATPSA can help instructors assess the impact of teaching reforms on students’ ability to solve traditional physics problems. Basic algebra and trigonometry are required to solve the problems. There is a range of difficulty in the ATPSA questions so that the test can assess students of varying levels, though the level of mathematics required for the questions does not change with the difficulty. There are no questions where a mathematical “trick” is needed. The questions on the ATPSA were created by the test developers.
Recommendations for choosing a problem-solving assessment
If you want to use a standardized method of scoring your students’ written solutions to your own physics problems and want to get a better sense of your students’ strengths and weaknesses with particular problem-solving skills, use the MAPS rubric. If you have a small number of students (undergraduate research students, graduate students, etc.,), you want to understand their problem-solving strengths and weaknesses in great depth, and you have time to individually go through the problem-solving exercise and associated questions with them, use the CAPS. If you want to assess your students’ problem-solving skills on textbook-like problems that cover Newton’s laws, momentum, and energy, want something that is standardized so that you can compare over time and to others, and is reasonably easy to score, use the ATPSA.
Add a Comment
