Skip to main content
SpringerLink
Log in

The use and misuse of journal metrics and other citation indicators

  • Varia - Scientometrics
  • Published:
Archivum Immunologiae et Therapiae Experimentalis Aims and scope

Abstract

This article reviews the nature and use of the journal impact factor and other common bibliometric measures for assessing research in the sciences and social sciences based on data compiled by Thomson Reuters. Journal impact factors are frequently misused to assess the influence of individual papers and authors, but such uses were never intended. Thomson Reuters also employs other measures of journal influence, which are contrasted with the impact factor. Finally, the author comments on the proper use of citation data in general, often as a supplement to peer review. This review may help government policymakers, university administrators, and individual researchers become better acquainted with the potential benefits and limitations of bibliometrics in the evaluation of research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Banks MA, Dellavalle R (2008) Emerging alternatives to the impact factor. OCLC Systems & Services 24(3). Available via: http://eprints.rclis.org/archive/00014614/01/OCLC2BPaper2BFinal-v2Feb2008.pdf

  • Bar-Ilan J (2008) Informetrics at the beginning of the 21st century: a review. J Informetrics 2: 1-52

    Article  Google Scholar 

  • Bensman SJ (2007) Garfield and the impact factor. Annu Rev Inf Sci Technol 41: 93-155

    Article  Google Scholar 

  • Bensman SJ (2007) The impact factor, total citations, and better citation mouse traps: a commentary. J Am Soc Inf Sci Technol 58: 1904-1908

    Article  Google Scholar 

  • Bergstrom CT (2007) Eigenfactor: measuring the value and prestige of scholarly journals. C&RL News 68:5. Available via: http://www.ala.org/ala/acrl/acrlpubs/crlnews/backissues2007/may07/eigenfactor.cfm , also see: http://www.eigenfactor.org//whyeigenfactor.htm

  • Bollen J, Rodriguez MA, Vande Sompel H (2006) Journal status. Scientometrics 69: 669-687

    Article  CAS  Google Scholar 

  • Borgman C, Furner J (2002) Scholarly communication and bibliometrics. Annu Rev Inf Sci Technol 36: 3-72

    Google Scholar 

  • Bornmann L, Mutz R, Neuhaus C et al (2008) Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics Sci. Environmental Politics 8: 93-102

    Article  Google Scholar 

  • Braun T, Glanzel W, Schubert A (2005) A Hirsch-type index for journals. Scientist 19:8

    Google Scholar 

  • Brin S, Page L (1998) The anatomy of a large-scale hypertextual web search engine. Comput Networks ISDN Systems 30: 107-117

    Article  Google Scholar 

  • Butler L (2008) Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework. Ethics Sci Environmental Politics 8: 83-92

    Article  Google Scholar 

  • Egghe L, Rousseau R (2002) A general framework for relative impact factors. Can J Inf Library Sci 21: 29-38

    Google Scholar 

  • Egghe L (2005) Continuous, weighted Lorenz theory and applications to the study of fractional relative impact factors. Inf Processing Management 41: 1330-1359

    Article  Google Scholar 

  • Egghe L (2006) Theory and practice of the g-index. Scientometrics 69: 131-152

    Article  Google Scholar 

  • Falagas ME, Kouranos VD, Arencibia-Jorge R et al (2008) Comparison of SCImago journal rank indicator with journal impact factor. FASEB J 22: 2623-2628

    Article  CAS  PubMed  Google Scholar 

  • Garfield E (1955) Citation indexes for science. A new dimension in documentation through association of ideas. Science 122: 108-111

    CAS  Google Scholar 

  • Garfield E (1972) Citation analysis as a tool in journal evaluation. Science 178: 471-479

    Article  CAS  PubMed  Google Scholar 

  • Garfield E (1976) Significant journals of science. Nature 264: 609-615

    Article  CAS  PubMed  Google Scholar 

  • Garfield E (1979) Citation Indexing: Its Theory and Application in Science, Technology, and Humanities. John Wiley & Sons, New York

    Google Scholar 

  • Garfield E (1996) How can impact factors be improved?. BMJ 313: 411-413

    CAS  PubMed  Google Scholar 

  • Garfield E (1999) Journal impact factor: a brief review. CMAJ 161: 979-980

    CAS  PubMed  Google Scholar 

  • Garfield E (2006) The history and meaning of the journal impact factor. JAMA 295: 90-103

    Article  CAS  PubMed  Google Scholar 

  • Garfield E, Sher IH (1963) New factors in the evaluation of scientific literature through citation indexing. Am Documentation 14: 195-201

    Article  CAS  Google Scholar 

  • Glanzel W (2008) Seven myths in bibliometrics: about facts and fiction in quantitative science studies. Proceedings of the WIS 2008, Berlin. Fourth International Conference on Webometrics, Informetrics and Scientometrics & Ninth COLLNET Meeting. Available via: http://www.collnet.de//Berlin-2008/GlanzelWIS2008smb.pdf

  • Glanzel W, Moed HF (2002) Journal impact measures in bibliometric research. Scientometrics 53: 171-193

    Article  Google Scholar 

  • Harnad S (2008) Validating research performance metrics against peer rankings. Ethics Sci. Environmental Politics 8: 103-107

    Article  Google Scholar 

  • Harzing A-WK, van der Wal R (2008) Google Scholar as a new source for citation analysis. Ethics Sci. Environmental Politics 8:61-73. Available via: http://www.harzing.com/resources.htm#/pop_hindex.htm

  • Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A 102: 16569-16572

    Article  CAS  PubMed  Google Scholar 

  • Hirsch JE (2007) Does the h-index have predictive power?. Proc Natl Acad Sci U S A 104: 19193-19198

    Article  CAS  PubMed  Google Scholar 

  • Jasco P (2005) As we may search: Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Curr Sci 89: 1537-1547

    Google Scholar 

  • Joint Committee on Quantitative Assessment of Research (2008) Citation Statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS), by Robert Adler, John Ewing (Chair), and Peter Taylor. Available via: http://www.mathunion.org/fileadmin/IMU//Report/CitationStatistics.pdf

  • Ketcham CM, Crawford JM (2007) The impact of review articles. Lab Invest 87: 1174-1185

    Article  CAS  PubMed  Google Scholar 

  • Krauze T (1977) The sociology of science in Poland. In: D. A. Pendlebury: Citation indicators use and misuse Merton RK, Gaston J (eds) The Sociology of Science in Europe. Southern Illinois University Press, Carbondale, Illinois, pp. 193-223

  • Leydesdorff L (2008) Caveats in the use of citation indicators in research and journal evaluations. J Am Soc Inf Sci Technol 59: 278-287

    Article  Google Scholar 

  • Markpin T, Boonradsamee B, Ruksinsut K et al (2008) Article-count impact factor of materials science journals in SCI database. Scientometrics 75: 251-261

    Article  CAS  Google Scholar 

  • Meho LI, Yang K (2007) A new era in citation and bibliometric analyses: Web of Science. Scopus, and Google Scholar. J Am Soc Inf Sci Technol 58: 2105-2125

    Article  Google Scholar 

  • Moed HF (2002) The impact-factors debate: the ISI’s uses and limits. Nature 415: 731-732

    CAS  PubMed  Google Scholar 

  • Moed HF (2005) Citation analysis of scientific journals and journal impact measures. Curr Sci 89: 1990-1996

    Google Scholar 

  • Moed HF (2005) Citation analysis in research evaluation. Springer, Dordrecht

    Google Scholar 

  • Moed HF, Glanzel W, Schmoch U (eds) (2004) Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems. Kluwer Academic Publishers, Dordrecht

  • Moed HF, van Leeuwen TN (1995) Improving the accuracy of Institute for Scientific Information’s journal impact factors. J Am Soc Inf Sci 46: 461-467

    Article  Google Scholar 

  • Moed HF, van Leeuwen TN (1996) Impact factors can mislead. Nature 381: 186

    Article  CAS  PubMed  Google Scholar 

  • Moed HF, van Leeuwen TN, Reedijk J (1996) A critical analysis of the journal impact factors of Angewandte Chemie and Journal of the American Chemical Society: inaccuracies in published impact factors based on overall citation counts only. Scientometrics 37: 105-116

    Article  Google Scholar 

  • Moed HF, van Leeuwen TN, Reedijk J (1999) Towards appropriate indicators of journal impact. Scientometrics 46: 575-589

    Article  Google Scholar 

  • Monastersky R (2005) The number that’s devouring science. The Chronicle of Higher Education 52: A12

    Google Scholar 

  • Narin F (1976) Evaluative bibliometrics: the use of publication and citation analysis in the evaluation of scientific activity. Computer Horizons Inc., Cherry Hill, New Jersey

  • Nicolaisen J (2007) Citation analysis. Annu Rev Inf Sci Technol 41: 609-641

    Article  Google Scholar 

  • Ossowska M, Ossowski S (1936) The science of science. Organon 1: 1-12

    Google Scholar 

  • Pinski G, Narin F (1976) Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics. Inf Processing Management 12: 297-312

    Article  Google Scholar 

  • Pudovkin AI, Garfield E (2004) Rank-normalized impact factor: a way to compare journal performance across subject categories. ASIST 2004: Proceedings of the 67th ASIS&T Annual Meeting 41:507-515

    Google Scholar 

  • Rousseau R (2002) Journal evaluation: technical and practical issues. Library Trends 50: 418-439

    Google Scholar 

  • Rousseau R (2005) Median and percentile impact factors. Scientometrics 63: 431-441

    Article  CAS  Google Scholar 

  • Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. BMJ 314: 498-502

    CAS  PubMed  Google Scholar 

  • Sombatsompop N, Markpin T (2005) Making an equality of ISI impact factors for different subject fields. J Am Soc Inf Sci Technol 56: 676-783

    Article  Google Scholar 

  • Sombatsompop N, Markpin T, Premkamolnetr N (2004) A modified method for calculating the impact factors of the journals in ISI Journal Citation Reports: polymer science category in 1997-2001. Scientometrics 60: 217-235

    Article  CAS  Google Scholar 

  • Sombatsompop N, Markpin T, Yochai W et al (2005) An evaluation of research performance for different subject categories using impact factor point average (IFPA) index: Thailand case study. Scientometrics 65: 293-205

    Article  CAS  Google Scholar 

  • Stringer MJ, Sales-Pardo M, Nunes Amaral LA (2008) Effectiveness of journal ranking schemes as a tool for locating information. PLoS One 3:e1683. Available via: http://www.plosone.org/article/info:doi2F10.13712Fjournal.pone.0001683

  • Thomson Reuters, http://scientific.thomson.com/products/jcr/

  • Thomson Reuters, http://scientific.thomsonreuters.com/products/jpi/

  • Thomson Reuters, http://scientific.thomson.com/products/esi/

  • Vaidya JS (2005) V-index: a fairer index to quantify an individual’s research output capacity. BMJ Rapid Response. Available via: http://bmj.bmjjournals.com/cgi/eletters/331//7528/1339-c#123188

  • van Leeuwen TN, Moed HF (2002) Development and application of journal impact measures in the Dutch science system. Scientometrics 53: 249-266

    Article  Google Scholar 

  • van Leeuwen TN, Moed HF (2005) Characteristics of journal impact factors: the effects of uncitedness and citation distribution on the understanding of journal impact factors. Scientometrics 63: 357-371

    Article  Google Scholar 

  • van Leeuwen TN, Moed HF, Reedijk J (1999) Critical comments on Institute for Scientific Information impact factors: a sample of inorganic molecular chemistry journals. J Inf Sci 25: 489-498

    Article  Google Scholar 

  • Vinkler P (2004) Characterization of the impact of sets of scientific papers: the Garfield (impact) factor. J Am Soc Inf Sci Technol 55: 431-435

    Article  CAS  Google Scholar 

  • Wouters P (1999) The Citation Culture. University of Amsterdam, Amsterdam, pp. 82-93

  • Zitt M, Bassecoulard E (2008) Challenges for scientometric indicators: data demining, knowledge-flow measurements and diversity issues. Ethics Sci. Environmental Politics 8: 49-50

    Article  Google Scholar 

  • Zitt M, Small H (2008) Modifying the journal impact factor by fractional citation weighting: the audience factor. J Am Soc Inf Sci Technol 59: 1856-1860

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Thomson Reuters, Philadelphia, PA, USA

    David A. Pendlebury

Authors
  1. David A. Pendlebury
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David A. Pendlebury.

About this article

Cite this article

Pendlebury, D.A. The use and misuse of journal metrics and other citation indicators. Arch. Immunol. Ther. Exp. 57, 1–11 (2009). https://doi.org/10.1007/s00005-009-0008-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00005-009-0008-y

Keywords

Search

Navigation