Bauer et al have published two articles on citation search:Bakkalbasi N, Bauer K, Glover J, Wang L (2006)
Three options for citation tracking: Google Scholar, Scopus and Web of Science Biomedical Digital Libraries, Vol. 3, No. 7, 29 June.
Bauer K and Bakkalbasi N (2005) An Examination of Citation Counts in a New Scholarly Communication Environment
D-Lib Magazine, Vol. 11, No. 9.
Noruzi also made some brief evaluations in an article:
Noruzi, Alireza Google Scholar: The New Generation of Citation Indexes
LIBRI Vol. 55, Iss. 4, p. 170-80
Belew K compared citation search in WoS and Google Scholar:
Belew, RK (2005) Scientific impact quantity and quality: [PDF] Analysis of two sources of bibliographic data.
The first article of Bauer and Bakkalbasi showed the citation count for GS was higher than WoS and Scopus for 2000. But for 1985 WoS seem to be best to cover citations. Comparing WoS and Scopus, WoS found more citations for 1985 but for 2000 it was similar.
The next article of Bauer, Bakkalbasi et al evaluated journal articles from two disciplines: oncology and condensed matter physics (CM physics) and two years: 1993 and 2003.
Their conclusion is:”This study did not identify any one of the three tools studied to be the answer to all citation tracking needs”. Scopus shows strength for oncology articles from 2003, but WoS performed better for CM physics and was better for both disciplines published in 1993. GS returned smaller number but had a large set of unique citing material for 2003. Bauer, Bakkalbasi et al make clear:”…it is clear that Google Scholar provides unique citing material.”
The article by Belew compares GS with WoS by author search. Belew randomly selected six academics from same interdisciplinary department and bibliographies of all publications by these authors were manually reconciled against 203 references found by one or both systems. WoS discovered 4741 citations and GS 4045, but when evaluating each author 2 authors get significantly more citations in GS.
Belew indicates that because of the quality in some bibliographic citations it’s common to find that same publication has been treated as more than one record. When searching cited ref search in WoS for an author you can find these types of errors. As Belew indicates in Table 1. With these types of errors it’s possible to loose citations for an article in WoS, but instead there are sometimes duplicates of an article (preprint and original article) that inflates citation count. Belew does not discuss that GS often shows duplicates and sometimes if you manually check the number of times cited it’s incorrect displayed.
Belew conclusion is: “GS seems competitive in terms of coverage for materials published in the last twenty years; before then WoS seems to dominate”.
We also earlier this year made a small test between Scopus and WoS by searching author name, but just author names that we can sort out as unique.
Noruzi made free text searches when testing citation search with search statement: webometrics OR webometric. Freetext search is not a proper subject search. As Bauer et al is pointing out WoS, GS and Scopus databases processes a freetext search in different ways. For example Google Scholar indexes even the fulltext of articles in contrary to WoS and Scopus. Though in this case Noruzi still have just compared each known article, though the method of choosing articles and the low amount of articles may be arguable.
None of Belew and Bauer et al have discussed the problems with citation counting in Google Scholar. Though Peter Jacso have criticized Bauer et al and presented examples of flaws in Google Scholar:
Jacso, Peter ([2005b]) Google Scholar and The Scientist
(Published on university homesite as extra material). [online] http://www2.hawaii.edu/~jacso/extra/gs/
I believe the percentage of flaws in Google Scholar may not decrease the value of the findings significantly of Belew and Bauer et al but it should be considered and discussed. Research on the propotions of citation counting flaws in Google Scholar would be of considerable value for future evaluations.
I checked the citation counting in Google Scholar of the first article Noruzi refers to in his test in Table 2:
C Almind and P. Ingwersen Informetric analyses on the world wide web Journal of Documentation Vol. 54 Iss. 4, p. 404-426
I received 4 hits where the first hit clusters 11 duplicates (look at link group of 11). 3 duplicates (hit 2-4) are unclustered. Together it’s 192 citations for the article of Almind et al. But if you check the reliance of citations in all hits you will find duplicates. I evaluated maually all 192 citations together and found 13 obvious duplicates. It’s manually checked and some more duplicates may be found. All records in chinese letters are not checked. Here are screenshots on all duplicates put together with an image editing software:
Of 192 citations from GS, 12 is duplicates which gives these results: GS 180, WoS 90. This means 6% is incorrect citations.
Of course WoS could have duplicates also.
Conclusion: Scopus is important for finding more citations from 1996 and current. Google Scholar is important because it finds a lot of unique citations but each reference with information on times cited should be manually checked by counting and looking for duplicates. Web of Science is still competetive, especially for older material.
As Noruzi mention in his article GS indexes a lot more of publication types and from various languages. If every citation, no matter from which source, has the same value of 100%, GS is an important source. The discussion should exceed on the value of each citation. Should self-citation get any value at all? Should articles not peer-reviewed get a lower value for their citations?