Abstract
Citation analysis is a useful method for studying a wide range of topics in bibliometrics and the sociology of science. However, many challenges have been made to the validity and reliability of the underlying assumptions, the data, and the methods used in citation studies. This article addresses these issues in three parts. First is a brief review of validity and reliability issues in citation research. Next we explore measurement error in a principal source of journal-to-journal citation data, the Institute for Scientific Information'sJournal Citation Reports. Possible sources of measurement error include discrepancies between citing and cited data, changed or deleted journal titles, aberrant abbreviations, and listing algorithms. The last section is a detailed description of ways to overcome some of the measurement errors. The data and examples are drawn from a journal-to-journal citation study in the fields of Communication, Information Science, and Library Science.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
J. BENIGER (1988). Information and communication: The new convergence.Communication Research, 15 (2), 198–218.
R. W. BOHANNON (1986). Letter to the editor.Physical Therapy, 66(9), 1431–1432.
C. L. BORGMAN & J. SCHEMENT (1988). Information science and communication science: An essay on convergence. In: PRENTICE, A. (Ed.),Information science as a discipline, (in press) NY: Schuman.
B. BOYCE & C. BANNING (1979). Data accuracy in citation studies. RQ, 18, 349–150.
R. BROADUS (1987). Early approaches to bibliometrics.Journal of the American Society for Information Science, 38(2), 127–129.
L. D. BROWN & J. C. GARDNER (1985). Using citation analysis to assess the impact of journals and articles on contemporary accounting research.Journal of Accounting Research, 23(1), 84–109.
R. S. BURT (1978). Stratification and prestige among elite experts in methodological and mathematical sociology circa 1975.Social Networks, 1, 105–158.
R. S. BURT & P. DOREIAN (1982). Testing a structural model of perception: Conformity and deviance with respect to journal norms in elite sociological methodology.Quality and Quantity, 16, 109–150.
A. R. BUSS & J. R. McDERMOTT (1976). Ratings of psychology journals compared to objective measures of journal impact.American Psychologist, 31(9), 675–678.
M. CARPENTER & F. NARIN (1973). Clustering of scientific journals.Journal of the American Society for Information Science, 23, 425–436.
H. CASON & M. LUBOTSKY (1936). The influence and dependence of psychological journals on each other.Psychological Bulletin, 33(2), 95–103.
A. CAWKELL (1978). Evaluating scientific journals with Journal Citation Reports: A case study in acoustics.Journal of American Society for Information Science, 29, 41–46.
D. CHUBIN (1983).Sociology of sciences: An annotated bibliography on invisible colleges, 1972–1981, NY: Garland Publishing.
J. COLE & S. COLE (1971). Measuring the quality of sociological research: Problems in the use of the Science citation index.American Sociologist, 6, 23–29.
D. CRANE (1972).Invisible colleges: Diffusion of knowledge in scientific communities. Chicago: University of Chicago Press.
D. CRANE (1974). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals.The American Sociologist, 2, 195–201.
L. CUMMINGS & P. FROST (Eds), (1985).Publishing in the organizational sciences. Homewood, ILL: Irwin.
J. DICKMAN (1984). How do secondary services select journals?Cleveland Clinic Quarterly, 51, 575–578.
J. DICKMAN & G. PLATEAU (1983). Foreign literature challenges to Chemical Abstracts Service.Journal of Chemical Information and Computer Science, 23(2), 70–73.
P. DOREIAN (1985). Structural equivalence in a psychology journal network.Journal of the American Society for Information Science, November, 411–417.
P. DOREIAN (1985). A measure of standing of journals in stratified networks.Journal of the American Society for Information Science, 8(5/6), 341–363.
P. DOREIAN & T. FARARO (1985). Structural equivalence in a journal network.Journal of the American Society for Information Science, 36(1), 28–37.
D. EDGE (1979). Quantitative measures of communication in science: A critical review.History of Science, 17, 102–133.
E. GARFIELD (1979).Citation indexing: Its theory and application in science, technology and humanities. NY: Wiley.
E. GARFIELD (1984). The 100 most-cited papers ever and how we select citation classics.Current Contents, 23, 3–9.
W. GARVEY (1979).Communication: The essence of science. Facilitating information exchange among librarians, scientists, engineers, and students. NY: Pergamon Press.
J. GOODRICH & C. ROLAND (1977). Accuracy of published medical reference citations.Journal of Technical Writing and Communication, 7, 15–19.
M. GORDON (1982). Citation ranking versus subjective evaluation in the determination of journal hierarchies in the social sciences.Journal of the American Society for Information Science, 33, 55–57.
W. HAGSTROM (1965).The scientific community. NY: Basic Books.
P. J. HART (1987).The use and role of commercial online databases in four organizations. Unpublished PhD dissertation. Los Angeles: University of Southern California, Annenberg School of Communications.
G. HIRST (1978). Discipline impact factors: A method for determining core journal lists.Journal of the American Society for Information Science, 29, 171–172.
R. HJERPPE (1980).A bibliography of bibliometrics and citationindexing and analysis. (TRITA-LIB-2013). Stockholm: Royal Institute of Technology Library.
Institute for Scientific Information. (1985).Journal Citation Reports. Philadelphia: Institute for Scientific Information.
D. LINDSEY (1978).The scientific publication system in social science. San Francisco: Josey-Bass.
M. LINE (1985). Changes in rank lists of serials over time: Interluding versus citation data.College & Research Libraries, 46(1), 77–79.
S. LOCK (1986).A difficult balance: Editorial peer review in medicine. Philadelphia, PA: Institute for Scientific Information.
T. LUCE & D. JOHNSON (1978). Rating of education and psychological journals.Educational Researcher, 7, 8–10.
K. MACE & H. WARNER (1973). Ratings of psychology journals.American Psychologist, 28(2), 184–186.
P. McALLISTER; R. ANDERSON & F. NARIN (1980). Comparison of peer and citation assessment of the influence of scientific journals.Journal of the American Society for Information Science, 31, 147–152.
N. MIDORIKAWA (1983). Citation analysis of physics journals: Comparison of subfield of physics.Scientometrics, 5(6), 361–374.
F. NARIN; M. CARPENTER & N. BERLT (1972). Inter-relationships of scientific journals.Journal of the American Society for Information Science, 23, 323–331.
F. NARIN & D. GARSIDE (1972). Journal relationships in special education.Exceptional Children, 38(9), 695–703.
W. NORD (1985). Looking at ourselves as we look at others: An exploration of the publication system for organizational research. In: L. CUMMINGS & P. FROST (Eds),Publishing in the organizational sciences. Homewood, ILL: Irwin.
W. PAISLEY (1984). Communication in the communication sciences. In: B. DERVIN & M. VOIGT (Eds),Progress in the communication sciences, Vol. 5 (pp. 1–43.) Norwood, NJ: Ablex.
D. PETERS & S. CECI (1982). Peer review practices of psychological journals: The fate of published articles, submitted again.Behavioral and Brain Science, 2(5), 187–195. (Cited inLock, 1985).
A. PORTER (1977). Citation analysis: Queries and caveats.Social Studies of Science, 7, 257–267.
W. POWELL (1985).Getting into print: The decision-making process in scholarly publishing. Chicago: University of Chicago Press.
B. REEVES & C. L. BORGMAN (1983). A bibliometric evaluation of core journals in communication research.Human Communication Research, 10(1), 119–136.
R. E. RICE; C. L. BORGMAN & B. REEVES (1988). Citation networks of communication journals, 1977–1985.Human Communicatiin Research, 15(2), 256–283.
J. RUSHTON (1984). The validity of citation counts.Bulletin of the British Psychological Society, 37, 33–36.
W. SATARIANO (1978). Journal use in sociology: Citation analysis versus readership patterns.Library Quarterly, 48(3), 293–300.
A. SINGLETON (1976). Journal ranking and selection: A review in physics.Journal of Documentation, 32(4), 258–289.
W. SCOTT (1974). Interreferee agreement on some characteristics of manuscripts submitted to theJournal of Personality and Social Psychology.American Psychologist, 29, 698–702. (Cited in Bohannon, 1986).
H. SMALL (1981). The relationship of information science to the social sciences: A co-citation analysis.Information Processing and Management, 17, 39–50.
J. SMART (1983). Perceived quality and citation rates of education journals.Research in Higher Education, 19(2), 175–182.
L. SMITH (1981). Citation analysis.Library Trends, Summer, 83–106.
N. SMITH & D. CAULLEY (1981). The evaluation of education journals through the study of citations.Educational Researcher, 10, 11–12.
C. SO (1988). Citation networks of communication journals.Human Communication Research, 15(2), 236–255.
SSCI Journal citation reports; a bibliometric analysis of social sciences journals in the ISI data base. Social Sciences Citation Index 1981 annual, vol. 7. (1981, 1984, 1985, 1986). Philadelphia: Institute for Scientific Information.
F. THORNE (1977). The Citation Index: Another case of spurious validity.Journal of Clinical Psychology, October, 33, 1157–1161.
Ulrich's International Periodicals Guide; A classified Guide to a Selected List of Current Periodicals, Foreign and Domestic. 25th ed, 1986–87. (1986). NY: R. R. Bowker.
H. WHITE & B. GRIFFITH (1982). Authors as markers of intellectual space: Co-citation on studies of science, technology and society.Journal of Documentation, 38(4), 255–272.
M. WHITE & K. WHITE (1977). Citation analysis of psychology journals.American Psychologist, 32, 301–305.
N. WHITMAN & B. EYRE (1985). The pattern of publishing previously rejected articles in selected journals.Family Medicine, 17(1), 26–28.
S. Jr. WIBERLEY (1982). Journal rankings from citation studies: A comparison of national and local data from social work.Library Quarterly, 52(4), 348–359.
M. E. WILLIAMS (Ed.), (1985).Computer-Readable Databases: A Directory and Sourcebook, Vols 1–2. Chicago: American Library Association.
L. ZHIGNESSE & C. OSGOOD (1967). Bibliographic citation characteristics of the psychological journal network in 1950 and 1960.American Psychologist, 22, 778–791.
J. ZIMAN (1968).Public knowledge. Cambridge, England: Cambridge University Press.
H. ZUCKERMAN & R. MERTON (1971). Patterns of evaluation in science: Institutionalization, structure, and functions of the referee system.Minerva, 9, 66–100. (Cited in BOHANNON 1986.)
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Rice, R.E., Borgman, C.L., Bednarski, D. et al. Journal-to-journal citation data: Issues of validity and reliability. Scientometrics 15, 257–282 (1989). https://doi.org/10.1007/BF02017202
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF02017202