Parameter estimation for peer grading under incomplete design
Educational and Psychological Measurement
Methodology developed in the paper supports administration and analysis of peer grading exercises. The technique is especially useful for the incomplete design case, wherein each grader does not grade each paper. The procedure produces simultaneous estimates of (a) bias of each grader, (b) “true” grade for each paper, and (c) relative consistency of each grader. The approach is based on a simple linear statistical model, augmented by a zerosum constraint on grader bias parameters. Ordinary least squares (OLS) estimates are computed and then refined via generalized least squares (GLS) estimation. Case weights for the GLS are based on consistency patterns of the graders, as revealed in the OLS step. The methodology is applied to artificial data to demonstrate its effectiveness. The methodology applies to any situation wherein multiple raters rate multiple ratees on an additive scale. © 1988, Sage Publications. All rights reserved.
Original Publication Date
DOI of published version
Wilson, Hoyt G., "Parameter estimation for peer grading under incomplete design" (1988). Faculty Publications. 4666.