Meeting of the Board of Regents | December 2010

SUMMARY

Issue for Discussion

What steps will the Department take to maintain the rigor of the Comprehensive English Regents Examination? What steps will the Department take in the future for test development to appropriately increase the rigor of the English Regents Examination?

Reason(s) for Consideration

            For Discussion

Proposed Handling

This Item will come before the P-12 Education Committee for discussion at the December 2010 meeting.

Background Information

In June 1999, the Department implemented a new design for the Comprehensive English Regents Exam; it was administered in two separate three-hour sessions over two days. This format created significant hardship for both students and administrators around the State. On three separate occasions, adverse weather conditions disrupted the administration of the assessment, and students who started the test were unable to complete it; they were forced to retest at the next available testing period (June). Superintendents and other administrators repeatedly requested that the State redesign the exam for one three-hour session. In February 2009, Department staff recommended to the EMSC Committee that we honor this request, and the Board directed staff to redesign the test and maintain a parallel level of rigor despite the loss of three hours of testing time. In November 2009, the Department released test specifications for the new assessment and informed the field that it would be administered in January 2011.

In July 2010, the Department reset cut scores on the Grades 3-8 Testing Program. This process included analyzing the relationship between Grade 8 ELA Exam performance and subsequent performance on the Comprehensive English Regents Exam. This was done to help strengthen the predictive validity of the Grade 8 assessment and to inform administrators, teachers, and parents as to whether or not students are on track for college and/or career readiness.  After the administration of the new exam, the Department will need to conduct a Standards Revisitation Process. The Standards Revisitation Process will ensure that the Department maintains the rigor associated with the original English Regents Examination from 1999. The Department is also working to study the college readiness scale score of 75 and plans to conduct additional analysis with the College Board.  An overview of the plan is presented in Appendix A. 

The Department is committed to providing a high quality, rigorous commencement level English testing program that is aligned to college and career readiness standards. The redesign of the English Regents Exam is a first step and this new test is only an interim measure.  The Department has committed to participate in the development of the PARCC Common Core assessments which will provide tests in grades 9-11 for English beginning in 2014-15. During the interim period, the Department is exploring two possible approaches for continued development of a rigorous, college and career ready English Regents Exam: 1) discontinue the constructed response questions for the Literary Analysis section and replace them with a full essay which may require extending the administration time; or 2) implementing a grades 9-11 English Testing Program that is linked to the current Grades ELA 3-8 Testing Program that will continue to provide year-by-year achievement and growth data for students, schools, districts, and the state.

Appendix A

Comprehensive English Regents Exam


Performance Standards Revisitation Plan

This revised exam will be based on the 2005 high school level English Language Arts Core Curriculum, Performance Indicators, and the Key Ideas of listening, reading, and writing.    

The revised exam retains the following components from the current exam:

• Learning Standards, Key Ideas, &  Performance Indicators
• One listening passage
• Critical Lens Essay Task
• Paired literature passages
• Inclusion of non-fiction reading materials

Despite changes to the format and composition of some item types, the development of the new exam focused on maintaining the validity and consistency around the content being assessed. Our test specifications, percentages of content to be assessed, will be consistent with the original test. Having fewer essays on a test does not necessarily make a test easier; it is the overall test difficulty combined with the placement of the cut score(s) that determines rigor. The Department is still requiring that students write a full essay to extract a valid writing sample; we will weight the writing score to compensate for fewer writing items. The original English Regents Examination had very high internal consistency for the four essays (0.89). We will evaluate the placement of the new cut scores and calculate the internal reliability of the new assessment to ensure that the rigor is comparable to the original test and through the equating process, ensure the inferences regarding student achievement remains consistent.

Assumptions

The performance Standards Revisitation Process is based on four assumptions:

1. Maintaining rigor: the expected rigor of the new cut scores on the exam should be comparable with the current cut scores applied to the Comprehensive English Regents Exam.
2. Alignment with the performance standards from Grades 3-8: continue to maintain the alignment between the adjusted standards for ELA Grades 3-8 and the new Exam.
3. Readiness for the next educational level: performance standards provide information about students’ readiness for their next educational level.
4. College readiness indicator: colleges have indicated that a scale score of 75 denotes readiness to enroll in credit bearing English classes.

Goals

There are three goals for the Standards Revisitation Process: 1) revisit the standards and requirements due to the new test format; 2) describe the framework for collecting evidence to support the interpretation of the cut scores; and 3) continue the process for systematically collecting evidence to evaluate the use of a scale score of 75 as an indicator of college readiness.

Methodology

There are three stages for the Standards Revisitation Process: 1) Pre-policy Measurement review; 2) Performance Standards Revisitation; and 3) Post-policy Measurement Review.

Pre-policy Measurement Review

The Pre-policy measurement review is designed to obtain recommendations for the expected percentage of New York students who would be classified in each achievement level: 0-64, 65-84, and 85-100. The committee, made up of members of advisory groups to the Department, will be asked to consider five guiding questions:

• Given the differences in the new format of the assessment, what type of differences in the percentage of students at each level does the panel expect?
• What percentage of students in each achievement level would the panel find acceptable on the newly formatted exam?
• What consistency is expected between the data from the current and new testing programs, given the consistency in content and standards requirements shown between the two exams?
• What type of consistency does the panel expect between ELA Grade 8 and the Comprehensive English Regents Exam?
• What consistency is expected between national data and New York results for this newly-formatted exam compared with the previously-formatted exam?

Following review of these guiding questions, panelists will review trend data from multiple assessment programs:

Standards Revisitation Process: Content Review/Instructional Impact

Content Review committees will meet in Albany to revisit the performance standards. They will study the changes in format to the test and evaluate whether the cut scores mapped to the newly-formatted exam continue to support the interpretations regarding student achievement. Given the assumption that the performance standards will provide information about student readiness for their next educational experience, college professors will participate in the process. The content experts will focus on consistency in content requirements for the cut scores across the current and newly-formatted tests.

It will be possible to identify where the original cut scores of 65 and 85 are located on the new exam. Panelists will use PLDs to focus on the items around the two cut scores. They will evaluate the knowledge and skills these items are measuring, how they relate to the standards and PLDs, and decide whether to keep the original cuts mapped onto the newly-formatted test or to move them. If they recommend moving the cut points, they will indicate which direction and how far. The exercise will be conducted in two rounds. The panelists will also be asked to provide their recommendations on what subsequent data collection and analysis can be conducted to provide ongoing validity evidence supporting the interpretations of the performance standards. As a separate step, the scale score of 75 will also be identified in the Ordered Item Booklet. Panelists will evaluate the skills and knowledge of the items around this cut score and discuss why the content surrounding this scale score supports the observation from that students have demonstrated the knowledge and skills necessary for college readiness as well impact on classroom instruction.

Post-policy Measurement Review

The Post-policy Measurement Review panel will meet after the completion of the standards revisitation meetings with the content experts. The same committee who participated in the Pre-policy Measurement Review will participate in this activity.  The purpose of the review panel is to integrate results from the Pre-policy Measurement Review and the two content committees (including the synthesis meeting). Results from Pre-policy Measurement Review and content expert recommendations on the Standards Revisitation will be presented and discussed. Following the review of the methods and results, panelists will be asked to independently integrate results from both meetings. They will then discuss their rationale for how they integrated the results.  Discussions regarding the scale score of 75 as an indicator of college readiness will occur. Further ideas for collecting validation evidence supporting interpretations from the three cut scores (65, 75, and 85) will be discussed and summarized in the technical report of the standards revisitation.

At the conclusion of all processes, a final report will be compiled and submitted to the Commissioner and Senior Deputy Commissioner for review; the Commissioner will make the final decision regarding the setting of the cut scores. The Department will post the score conversion charts by January 28, 2010.