Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests Book

Preface     xi
Fundamentals of Standard Setting     1
Contemporary Standard Setting: An Enduring Need     5
The Need to Make Decisions     6
The Benefits of Standard Setting     8
General Approaches to Standard Setting     9
Current Contexts for Standard Setting     11
What Is Standard Setting?     13
Kinds of Standards     14
Definitions of Standard Setting     14
Policy Issues and Standard Setting     19
Scoring Models     20
Research on Standard Setting     22
Rounding     23
Classification Errors     25
Item Scoring Criteria and Total-Test Performance Standards     29
Conclusions     33
Common Elements in Setting Performance Standards     35
Purpose     36
Choosing a Standard-Setting Method     41
Performance Level Labels     44
Performance Level Descriptions     46
Key Conceptualizations     48
Selecting and Training Standard-Setting Participants     49
Providing Feedback to Participants     53
Normative Feedback     54
Reality Feedback     55
Impact Feedback     56
Professional Guidelines for Standard Setting     57
Evaluating Standard Setting     59
Conclusions and a Foreword     63
Standard-Setting Methods     65
The Nedelsky Method     69
Procedures for the Nedelsky Method     70
Alternative Procedures and Limitations     71
The Ebel Method     75
Procedures for the Ebel Method     75
Alternative Procedures and Limitations     78
The Angoff Method and Angoff Variations     81
Procedures for the Angoff Method     82
Procedures for Angoff Variations     87
The Extended Angoff Method     87
The Yes/No Method     88
Alternative Procedures and Limitations     92
The Direct Consensus Method     97
Procedures for the Direct Consensus Method     98
Alternative Procedures and Limitations     102
The Contrasting Groups and Borderline Group Methods     105
Procedures for the Contrasting Groups Method     106
An Example Using the Contrasting Groups Method     108
The Borderline Group Method     112
Alternative Procedures and Limitations     113
The Body of Work and Other Holistic Methods     117
The Judgmental Policy Capturing Method     117
The Dominant Profile Method     120
The Analytical Judgment Method     121
Summary Analysis of Three Holistic Methods     122
Overview of the Body of Work Method     123
Procedures for the Body of Work Method     124
Training     125
Rangefinding     125
Pinpointing     129
Calculating Cut Scores     129
Interpreting the Logistic Regression Output     135
An Application of the Body of Work Method     138
Selecting Work Samples     139
Training Participants     140
Rangefinding     140
Alternative Procedures and Limitations     148
The Bookmark Method     155
Overview of the Bookmark Method     157
The Ordered Item Booklet     160
The Response Probability (RP) Value     162
Response Probabilities and Ordered Item Booklet Assembly-Rasch Model     162
Response Probabilities and Ordered Item Booklet Assembly-2PL Model     167
Directions to Bookmark Participants     172
Calculating Bookmark Cut Scores     176
An Implementation of the Bookmark Procedure     177
Training     177
Introducing the Ordered Item Booklet     179
Round One of a Bookmark Procedure     180
Obtaining Preliminary Bookmark Cut Scores     181
A Caveat and Caution Concerning Bookmark Cut Scores     184
Round One Feedback to Participants     185
Round Two of a Bookmark Procedure     186
Round Three of a Bookmark Procedure     187
Alternative Procedures and Limitations     189
The Item-Descriptor Matching Method     193
Procedures for the IDM Method     194
Alternative Procedures and Limitations     202
The Hofstee and Beuk Methods     207
The Hofstee Method     209
Procedures for Implementing the Hofstee Method     209
The Beuk Method     212
Procedures for Implementing the Beuk Method     212
Alternative Procedures and Limitations     214
Challenges and Future Directions in Standard Setting     217
Scheduling Standard-Setting Activities     219
Scheduling Standard Setting for Educational Assessments     219
Overall Plan     222
Participants     225
Materials      231
Final Preparations     234
At the Standard-Setting Site and Following Up     236
Scheduling Standard Setting for Credentialing Programs     237
Overall Plan     238
Participants     242
Materials     243
Final Preparations     244
At the Standard-Setting Site and Following Up     244
Conclusions and Recommendations     246
Vertically-Moderated Standard Setting     249
The Interrelated Challenges     250
A Brief History of Vertically-Moderated Standard Setting     253
What Is VMSS?     254
Approaches to VMSS     256
Applications of VMSS     257
An Illustration of VMSS Procedures     262
The Assessment Context     262
Preparing to Implement a VMSS Approach     263
Training VMSS Participants     264
Facilitating the VMSS Standard-Setting Meeting     265
Vertical Articulation of Cut Scores     266
Final Review, Adoption, and Conclusions     270
Alternative Procedures and Limitations     271
Standard Setting on Alternate Assessments     275
The Unique Challenges of Alternate Assessment     276
Necessary Conditions for Alternate Assessment Systems     278
A Generalized Holistic Method     283
Overview of an Application of the GH Method     285
Procedures for Implementing the GH Method     286
Conclusions and Discussion Regarding the GH Method     292
Alternative Procedures and Limitations     293
Special Topics and Next Steps     297
Rounding     298
Methods of Adjusting Cut Scores     299
Deciding How to Incorporate Uncertainty     302
Generalizability of Standards     306
Decision Consistency and Decision Accuracy     307
A Demonstration of Computing Decision Consistency and Decision Accuracy on Complex Tests     313
Other Decision Consistency Procedures     315
Summary and Future Directions     317
Using Multiple Methods of Standard Setting     319
Improving Participant Training     320
References     323
Glossary     333
Author Index     341
Subject Index     345
About the Authors     351