Proceedings of the AoIR-ASIST 2004 Workshop on Web Science Research Methods
Using Web Citations to Measure Scientific Journal Impact
Debora Shaw, School of Library and Information Science, Indiana University, 1320 E. 10th Street, Main Library 011, Bloomington, IN 47405-3907, U.S.A. shawd @ indiana.edu
Liwen Vaughan, Faculty of Information and Media Studies, University of Western Ontario, London, Ontario, N6A 5B7, Canada lvaughan @ uwo.ca
Entirely Web-based publication of scientific research, an alluring prospect, may change entirely the nature of scientific communication, but is there evidence that Web access can affect assessments of journal impact even in our predominantly print-on-paper age? Could Web citation data supplement or replace ISI-based measures of impact? Our studies show that Web citation data demonstrate wider geographic influence of journal articles and that these citations correlate with impact assessments using ISI data, especially in disciplinary journals.
Building on to our earlier study of LIS journals, we extended the scope of investigation to four science fields: biology, genetics, medicine, and multidisciplinary sciences (for example, Nature, Science) to validate earlier findings and to examine disciplinary difference. The inclusion of wider varieties of journals from various countries allowed an exploration of geographical and cultural impact manifested on different media (the Web vs. traditional media) of scientific communication.
We conducted Google searches for Web citations (text citations of, not links to) journal articles in the four fields. We recorded the number and nature of the items retrieved, and then compared the number of Web citations with the ISI citation counts. For a sample of 5,972 articles published in 114 journals, the median Web citation counts per journal article range from 6.2 in medicine to 10.4 in genetics; median ISI citation counts range from 1.4 for the multidisciplinary journals to 6.6 in genetics. It is clear that actual numbers of Web citations are higher than ISI citation counts for all four disciplines, suggesting that Web citations could provide more fine-grained assessment of impact. In most cases, the number of ISI and Web citations is correlated, indicating they are measuring the same thing.
We narrowed the focus of the Google results, considering only Web citations from research papers or class readings lists (these sources we took to represent intellectual impact, as opposed to the perfunctory listing, for example, of an article on a journal’s home page or an author’s c.v.). Even when we look at only the Web citations which represent intellectual impact, we find that ISI and Web citation counts are correlated. The percentage of Web citations indicating intellectual impact is about 30% for each area. Journals receiving more Web citations also have higher percentages of citations indicating intellectual impact (e.g., 50% of Web citations to Nature articles are of this type).
However, there are significantly lower correlations between Web and ISI citations to articles in journals published outside the UK or USA. The Web citations are generally more numerous than the ISI citation counts would predict. This suggests that Web citations to non-UK/USA journals may be more frequent in sources not covered by the ISI; i.e., the Web’s wider linguistic and geographic coverage reveals significantly more citations, coming from different sources. Thus, Web citations may provide a more global assessment of impact — the decentralised model to counter the centralisation of ISI lamented, for example, in the Valparaiso Declaration for Improved Scientific Communication in the Electronic Medium.
The multidisciplinary science journals show lower correlation between Web and ISI citation counts (45% of these journals show statistically significant correlation, compared with over 60% for the other disciplines). There is considerable variability in this category, with well-known multidisciplinary journals (such as Nature and Science) showing higher correlations (correlation coefficient 0.7 and 0.6 respectively), but lower correlations for the numerous journals published outside the UK/USA or available only in print.
As the dominance of print-based dissemination of scientific information is challenged, the Web is increasing the number and diversity of voices assessing contributions to the literature. It is reassuring that the Web-derived assessments generally correlate with the judgments from established evaluative techniques, notably ISI citation counts and journal impact factors. This observation would seem to indicate that Web citation counts could supplement or replace ISI measures of impact and could provide an increased diversity in coverage. However, we caution that the correlations between Web and ISI citations may be a transient phenomenon. The ease of self-publishing and self-citing on the Web may allow knowing authors to take advantage of the absence of review mechanisms to inflate their citation counts. The ongoing debate about the nature and ethics of self-citation can compound itself in many ways when the constraints of editorial and peer review are removed.