Meeting of the Board of Regents | November 2003
|
|
THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY, NY 12234 |
|
TO: |
The Honorable the Members of the Board of Regents |
|
FROM: |
James A. Kadamus |
|
COMMITTEE: |
EMSC-VESID |
|
TITLE OF ITEM: |
Physics Regents Examination |
|
DATE OF SUBMISSION: |
November 5, 2003 |
|
PROPOSED HANDLING: |
Discussion |
|
RATIONALE FOR ITEM: |
Implementation of Regents Policy |
|
STRATEGIC GOAL: |
Goals 1 and 2 |
|
AUTHORIZATION(S): |
SUMMARY:
The attached report, with accompanying description of the new cut score methodology for the 2004 physics examination and beyond and for the 2002 and 2003 physics examination (Attachment A), identifies actions being taken by the Department to implement the recommendations of the Regents Work Group on Physics.
Attachments
ACTIONS TO IMPLEMENT THE RECOMMENDATIONS OF THE REGENTS WORK GROUP ON PHYSICS
At your October meeting, you accepted the policy recommendations of the Regents Work Group on Physics concerning future examinations. In November, new passing (65) and passing with distinction (85) scores will be determined through a standard-setting process, consistent with the recommendations of the Work Group. These scores will be used to create a new conversion chart for the physical setting - physics Regents examination for January 2004 and beyond. This standard setting will set a new performance level that will redefine the Regents physics course as a "first-level" high school course in physics, comparable to other "first-level" high school sciences. This examination can and should be mastered by the average student, not only those in honors and advanced placement classes.
Consistent with these recommendations, a Standard Setting Committee will convene on November 5-6. The Committee will have teacher representation from all areas of the State as well as representatives of higher education and industry. The Committee�s work will generate the new passing and passing with distinction scores, mentioned above, which will be used for the January 2004 examination and beyond.
The Regents also asked the Commissioner to provide a proposal in November for addressing the test scores for students who took the physical setting - physics Regents examination in 2002 and 2003.
We propose that once the new cut scores have been established prospectively we use them to issue new score tables for the June and August 2002 and January and June 2003 examinations. These revised score tables would reflect how the passing scores would have been set for a first-level course, consistent with the Regents Work Group on Physics recommendations and the work of the Standard Setting Committee. These score tables would be issued by December 2003. (Additional details on this process are provided in Attachment A.)
School districts would be required to put the revised score on the student�s permanent record. However, districts would have discretion in how they use these scores for college admission transcripts, final class grades, class ranks, etc. This approach responds to concerns raised by districts regarding scores for students who graduated in 2002 and 2003. Districts have indicated that they do not want to reevaluate class grades and ranks for students who have already graduated and would like to record new grades as pass/fail. The Department will issue guidance to the field regarding local flexibility on the use of the physics examination grades.
Attachment A
Methodology to Determine New Cut Scores for 2004 Physics Examination and Beyond
A Standard Setting Committee of teachers, administrators, and postsecondary subject matter specialists will be convened to consider the characteristics of a first-level physics course and the related requirements for mastering the desired body of knowledge and skills. This Committee will be given a broad definition of what it means to achieve the State learning standards and what it means to achieve the State learning standards with distinction.
The Committee will spend time being more specific in defining particular knowledge and skills needed to meet the State learning standards in physics. Toward this end, much of the work will occur in large group discussions. When consensus is achieved on the definitions, Committee members will take the Regents physics examination. When taking the examination, committee members will be asked to pay particular attention to the knowledge and skills required to answer each question correctly or to achieve each point level for multi-point items.
After the experience of taking the examination and scoring the responses, the Committee will discuss which test questions must be answered correctly, consistent with their definition of achieving the State learning standards, and which questions are beyond the requirement for achieving the learning standards, consistent with the definitions related to a first-level course.
The Committee will then be involved in evaluations of ordered item booklets. The booklets contain "live" physics questions, with questions ordered from least to most difficult. Based on the experience of taking the test and of forming definitions of passing and of passing with distinction, the Committee will begin to form judgments about which items are necessary for students to answer correctly to achieve each of those two score points. With each round of these judgments, the Committee�s judgments will be analyzed and feedback will be provided to the Committee. The Committee will be given the opportunity to adjust their judgments, as necessary.
At the end of the process, the Committee will be given information concerning the impact of their decisions. The proportion of field-tested students that would have passed the examination or passed it with distinction will be provided. To comply with the Regents request that the physics examination be consistent with the other science examinations, information will also be provided concerning the relative difficulty of the test and the passing and passing with distinction score points. With this information, the Committee will be invited to modify in their judgments, as they deem necessary.
The scores representing passing and passing with distinction will be identified as the average difficulty value of the most difficult item, i.e., the breaking point between the items needed to achieve each of these two levels, the items beyond what is needed. These values will then be transformed to represent the 65 and 85 scale scores on a 0 � 100 scale.
Methodology to Determine New Cut Scores for 2002 and 2003 Physics Examinations
The Standard Setting Committee will establish standards for passing and for passing with distinction. To accomplish this task, the Committee will have identified test questions that should be answered correctly to achieve each of the standards. Each question, in turn, has a numeric difficulty value that is measured on the same difficulty scale used for all physics examinations, beginning in June 2002. That is, the difficulty values have been equated.
Consequently, the values corresponding to 65 and to 85 on the new adjusted (2004 and beyond) scale can be determined based on the equated item difficulty values. Because these values also correspond to the unadjusted scale scores, the new standards for passing and for passing with distinction can be applied both to the unadjusted scale values and to the new scale values. Following is an example of how it would work:
After the new standard setting, an item with a difficulty value of 0.7 may correspond to a scale value of 83. The same 0.7 item difficulty value would have corresponded on the June 2003 form to a scale value of 76. Therefore, an old scale score of 76 would now correspond to a new scale score of 83. A conversion table can be prepared showing that an old scale score of 76 becomes a new scale score of 83.
Using this methodology, we will create a conversion table(s) that can be used to give new scale value for each value of the former, unadjusted Regents examination scales, used for the June 2002 through June 2003 physics examinations.