Abstract
Previous research on the generalizability of student ratings of instruction has raised questions about the effects of academic discipline and item types on the generalizability of these data for making relative decisions about instructors and about courses. In particular, although student evaluation data appear to provide a reasonable basis for making decisions about instructors when generalizing across courses and students, when course is the object of measurement, the data appear to be less generalizable. It was suggested in the literature that this may be due to the type of evaluation items used or it may be due to academic discipline differences in the type of courses selected for study. This study used Biglan's (1973a) model for classifying disciplines along the dimensions of paradigmatic/preparadigmatic (hard/soft) and pure/applied. A nested sampling procedure yielded two sample types: courses within teachers, in which individual instructors taught more than one course; and teachers within courses, in which individual courses were taught by more than one instructor. For each sample type, evaluation forms for twenty courses within each discipline classification were sought. The evaluation items for this study were classified as measuring six dimensions of instruction: organization, breadth of coverage, group interaction, enthusiasm, grading, and individual rapport. Generalizability and decision studies were conducted in which, for one sample, teacher was the object of measurement, and for the second sample, course was the object of measurement. Results indicated that reliable decisions about instructors could reasonably be made from all six of the evaluation dimensions; however, reliability for course decisions varied greatly with the evaluation dimension, being highest for breadth of coverage and lowest for grading. The same general pattern was noted for the paradigmatic disciplines and the preparadigmatic-applied disciplines but not for the preparadigmatic-pure disciplines. It is suggested that a single evaluation instrument may not be uniformly applicable to all discipline areas.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Barnes, M. W., and Patterson, R. H. (1988, August). Using teaching evaluation results to plan department personnel strategies to accomplish the institutional teaching mission. Paper presented at the annual meeting of the Society for College and University Planning, Toronto.
Biglan, A. (1973a). The characteristics of subject matter in different academic areas.Journal of Applied Psychology 57(3): 195–203.
Biglan, A. (1973b). Relationships between subject matter characteristics and the structure and output of university departments.Journal of Applied Psychology 57(3):204–213.
Brennan, R. L. (1983).Elements of Generalizability Theory. Iowa City: The American College Testing Program.
Cashin, W. E. (1990). Students do rate different academic fields differently. In M. Theall and J. Franklin (eds.),Student Ratings of Instruction: Issues for Improving Practice. New Directions for Teaching and Learning 43: 113–121. San Francisco: Jossey-Bass.
Creswell, J. W., and Bean, J. P. (1981). Research output, socialization, and the Biglan model.Research in Higher Education 15:69–92.
Creswell, J. W., Seagren, A. T., and Henry, T. C. (1979). Professional development of training needs of department chairpersons: A test of the Biglan model.Planning and Change 10: 224–237.
Feldman, K. A. (1978). Course characteristics and college students' ratings of their teachers: What we know and what we don't.Research in Higher Education 9: 199–242.
Feldman, K. A. (1989). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies.Research in Higher Education 30: 583–645.
Gillmore, G. M. (1980, April). Student instructional ratings: To what universe can we dependably generalize results? Paper presented at the annual meeting of the American Educational Research Association, Boston.
Gillmore, G., Kane, M. T., and Naccarato, R. W. (1978). The generalizability of student ratings of instruction: Estimation of the teacher and course components.Journal of Educational Measurement 15(1): 1–13.
Malaney, G. D. (1986). Characteristics of graduate students in Biglan areas of study.Research in Higher Education 25: 328–341.
Marsh, H. W. (1981). The use of path analysis to estimate teacher and course effects in student ratings of instructional effectiveness.Applied Psychological Measurement 6: 47–60.
Marsh, H. W. (1984). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility.Journal of Educational Psychology 76: 707–754.
Muffo, J. A., and Langston, I. W. (1981). Biglan's dimensions: Are the perceptions empirically based?Research in Higher Education 15(2): 141–159.
Neumann, Y., and Neumann, L. (1983). Characteristics of academic areas and students' evaluation of instruction.Research in Higher Education 19(3): 323–334.
Neumann, L., and Neumann, Y. (1985). Determinants of students' instructional evaluation: A comparison of four levels of academic areas.Journal of Educational Research 78(3): 152–158.
Smart, J. C., and Elton, C. F. (1975). Goal orientations of academic departments: A test of Biglan's model.Journal of Applied Psychology 60(5): 580–588.
Smart, J. C., and Elton, C. F. (1976). Administrative roles of department chairmen. In J. C. Smart and J. R. Montgomery (eds.),Examining Departmental Management: New Directions for Institutional Research. San Francisco: Jossey-Bass.
Smart, J. C., and Elton, C. F. (1982). Validation of the Biglan model.Research in Higher Education 17: 213–229.
Smith, P. L. (1979). The generalizability of student ratings of courses: Asking the right questions.Journal of Educational Measurement 16(2): 77–87.
Stoecker, J. L. (1991, April). The Biglan classification revisited. Paper presented at the annual meeting of the American Educational Research Association, Chicago. ED#331442.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Barnes, L.L.B., Barnes, M.W. Academic discipline and generalizability of student evaluations of instruction. Res High Educ 34, 135–149 (1993). https://doi.org/10.1007/BF00992160
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF00992160