Google Scholar

This quite recently published article examines subject coverage in Google Scholar:

Neuhaus, Chris, (2006) Ellen Neuhaus, Alan Asher and Clint Wrede The Depth and Breadth of Google Scholar: An Empirical Study
Portal: Libraries and the Academy Vol. 6, No. 2, pp. 127-141.

From 47 databases 50 article titles were randomly collected from each database and compared with Google Scholar. Look at the figures here:

Conclusions: Each databse of 47 (with 2350 randomly selected articles) had a median and average coverage of 60%.

Neuhaus et al means GS weaknesses in subject coverage are social science and humanities databases and strenghts science and medical databases, open access databases, and single publisher databases.

Google Scholar has released an option for searching Related Articles, similar to PubMed Related Articles and other databases. Google Scholar is not using a thesaurus as PubMed do, but in advanced search you can limit to 7 broad subjects. With these Related Articles option you can try to search an article of your topic. Maybe you know the title or the author. When you find the article click on the link beneath it. Check this article reference by Judit Bar-Ilan about search engines evaluation:

Clicking the Related Articles link returns 99 related articles and one is for example Greg Notess’ old (yes, 2000 is old in the search engine evaluation world!) article about Search engines inconsistencies from the magazine Online.

I also tried some other smaller subjects within information science and it returns remarkably relevant hits. Nice satisfying option, but my brief evaluations doesn’t say anything about coverage of course. When trying to subject search Google Scholar (which as I said is not easy to do comprehensive) try to use the related articles, beside of free-text searching.

Bauer et al have published two articles on citation search:Bakkalbasi N, Bauer K, Glover J, Wang L (2006)
Three options for citation tracking: Google Scholar, Scopus and Web of Science Biomedical Digital Libraries, Vol. 3, No. 7, 29 June.

Bauer K and Bakkalbasi N (2005) An Examination of Citation Counts in a New Scholarly Communication Environment
D-Lib Magazine, Vol. 11, No. 9.

Noruzi also made some brief evaluations in an article:

Noruzi, Alireza Google Scholar: The New Generation of Citation Indexes
LIBRI Vol. 55, Iss. 4, p. 170-80
Belew K compared citation search in WoS and Google Scholar:

Belew, RK (2005) Scientific impact quantity and quality: [PDF] Analysis of two sources of bibliographic data.

The first article of Bauer and Bakkalbasi showed the citation count for GS was higher than WoS and Scopus for 2000. But for 1985 WoS seem to be best to cover citations. Comparing WoS and Scopus, WoS found more citations for 1985 but for 2000 it was similar.

The next article of Bauer, Bakkalbasi et al evaluated journal articles from two disciplines: oncology and condensed matter physics (CM physics) and two years: 1993 and 2003.

Their conclusion is:”This study did not identify any one of the three tools studied to be the answer to all citation tracking needs”. Scopus shows strength for oncology articles from 2003, but WoS performed better for CM physics and was better for both disciplines published in 1993. GS returned smaller number but had a large set of unique citing material for 2003. Bauer, Bakkalbasi et al make clear:”…it is clear that Google Scholar provides unique citing material.”

The article by Belew compares GS with WoS by author search. Belew randomly selected six academics from same interdisciplinary department and bibliographies of all publications by these authors were manually reconciled against 203 references found by one or both systems. WoS discovered 4741 citations and GS 4045, but when evaluating each author 2 authors get significantly more citations in GS.

Belew indicates that because of the quality in some bibliographic citations it’s common to find that same publication has been treated as more than one record. When searching cited ref search in WoS for an author you can find these types of errors. As Belew indicates in Table 1. With these types of errors it’s possible to loose citations for an article in WoS, but instead there are sometimes duplicates of an article (preprint and original article) that inflates citation count. Belew does not discuss that GS often shows duplicates and sometimes if you manually check the number of times cited it’s incorrect displayed.

Belew conclusion is: “GS seems competitive in terms of coverage for materials published in the last twenty years; before then WoS seems to dominate”.

We also earlier this year made a small test between Scopus and WoS by searching author name, but just author names that we can sort out as unique.

Noruzi made free text searches when testing citation search with search statement: webometrics OR webometric. Freetext search is not a proper subject search. As Bauer et al is pointing out WoS, GS and Scopus databases processes a freetext search in different ways. For example Google Scholar indexes even the fulltext of articles in contrary to WoS and Scopus. Though in this case Noruzi still have just compared each known article, though the method of choosing articles and the low amount of articles may be arguable.

None of Belew and Bauer et al have discussed the problems with citation counting in Google Scholar. Though Peter Jacso have criticized Bauer et al and presented examples of flaws in Google Scholar:

Jacso, Peter ([2005b]) Google Scholar and The Scientist
(Published on university homesite as extra material). [online]
I believe the percentage of flaws in Google Scholar may not decrease the value of the findings significantly of Belew and Bauer et al but it should be considered and discussed. Research on the propotions of citation counting flaws in Google Scholar would be of considerable value for future evaluations.

I checked the citation counting in Google Scholar of the first article Noruzi refers to in his test in Table 2:

C Almind and P. Ingwersen Informetric analyses on the world wide web Journal of Documentation Vol. 54 Iss. 4, p. 404-426

Check tihis screenshot:

I received 4 hits where the first hit clusters 11 duplicates (look at link group of 11). 3 duplicates (hit 2-4) are unclustered. Together it’s 192 citations for the article of Almind et al. But if you check the reliance of citations in all hits you will find duplicates. I evaluated maually all 192 citations together and found 13 obvious duplicates. It’s manually checked and some more duplicates may be found. All records in chinese letters are not checked. Here are screenshots on all duplicates put together with an image editing software:

Of 192 citations from GS, 12 is duplicates which gives these results: GS 180, WoS 90. This means 6% is incorrect citations.

Of course WoS could have duplicates also.

Conclusion: Scopus is important for finding more citations from 1996 and current. Google Scholar is important because it finds a lot of unique citations but each reference with information on times cited should be manually checked by counting and looking for duplicates. Web of Science is still competetive, especially for older material.

As Noruzi mention in his article GS indexes a lot more of publication types and from various languages. If every citation, no matter from which source, has the same value of 100%, GS is an important source. The discussion should exceed on the value of each citation. Should self-citation get any value at all? Should articles not peer-reviewed get a lower value for their citations?

When searching this article An Examination of Citation Counts in a New Scholarly Communication Environment by K Bauer, N Bakkalbasi – D-Lib Magazine, 2005 in Google Scholar you get the result cited by 8 other papers:

When checking all this 8 papers cited the Bauer et al article you well get three citations from the same source in The Scientist “The future of citation analysis”:

Google Scholar have managed to find 4 duplicates (as you can see in the last reference in link “group of 4”) and cluster them but missed two other duplicates.

I've made a page with a list of conference presentations reviewing Scopus and Google Scholar and also Web of Sceince in comparison with the former databases. Three examples follow below:

Jacso, Peter "The Endangered Database Species: Are the traditional commercial indexing/abstracting & full-text databases dead?"[PPT]
UK Serials Group 29th UKSG Annual Conference and Exhibition, University of Warwick 3-5 April 2006.

Jenkins, JR "Article Linker Integration with Google Scholar (or Google Scholar as referring source)"[PPT]
OpenURL and Metasearch: New Standards, Current Innovations, and Future Directions, September 19, 20, 21, 2005, Washington, DC.

Tarantino, Ezio "Scopus, WOK, Google Scholar: too much or not too much?" [PPT]
The International Coalition of Library Consortia Autumn 2005 7th European Meeting, Poznan, Poland 28.09 – 01.10, 2005.

If you have more suggestions, just post a comment. Please note, however, that it should be a conference presentation (i.e., not a lecture) and it should focus on some of the above-mentioned sources.

We made some citation frequency comparisons between Scopus, Web of Science and Google Scholar. As Scopus counts citations from 1996 we limited the comparisions to articles published from 1996 and current. The result of the figures in the screenshot showed:

Scopus finds 9% more citations than Web of Science when limited to articles from 1996-.

Scopus finds 20% more citations than Google Scholar when limited to articles from 1996-.

Web of Science finds 10% more citations than Google Scholar when limited to articles from 1996-.

Important to know is that Web of Science indexes more than 9,000 journals compared to Scopus 15,000 journals, though Web of Science argues that (according to Bradford's Law) they have the core journals which have the most citations. Google Scholar has no list of journals and other sources they index, but they index both articles from the proprietary web and scholarly archives, master theses, books etc. Google Scholar citation counting is not working properly either as we already pointed out in a previous posting. In this test all cited references from Scopus haven't been retrieved, just the indexed articles.

As we also already mentioned, the article "An Examination of Citation Counts in a New Scholarly Communication Environment" published in D-Lib magazine September 2005 Vol. 11, No. 9. by Kathleen Bauer et al at Yale University Library made some citation counting. But when we just counted all citations for a random 5 set of authors at Umeå university, Bauer et al made comparisons of the average number of times an article is cited. Both our test and the test by Bauer et al didn't check the Google Scholar inconsistencies of citations counting and duplicates.

Some of the findings from the article by Bauer et al follow below. The information derives from the tables in their article.

The search for articles published 2000 in Journal of the American Society for Information Science and Technology (JASIST) showed for example:

Web of Science counts 0.3 more citations than Scopus.

The search for articles published 1985 in Journal of the American Society for Information Science and Technology (JASIST) showed for example:

Web of Science counts 11.9 more citations than Scopus.

Because Scopus just count citations from articles published from 1996 and current the 11.9 difference is not surprising. Though the 0.3 difference for articles published from 2000 is more questionable. This test by Bauer et al has its limitations because it's limited to just one journal (i.e., JASIST).

Conclusion: Different testing methods at least shows that Scopus definitely is important when searching citations for articles published from 1996. Due to inconsistencies in Google Scholar its not suggested as a single usable tool for citation search.

In several of his writings, Peter Jacso has indicated the inconsistencies of Google Scholar. One important flaw is the citation search. Both his web published paper "Google Scolar and The scientist" and the article "As we may search" published in Current Science 2005 (please see References to literature) discuss the problems.

My testings indicates less inconsistencies than before, but still they exist. The article "An Examination of Citation Counts in a New Scholarly Communication Environment" published in D-Lib magazine September 2005 Vol. 11, No. 9. by Kathleen Bauer et al at Yale University Library made some comparisions of the average number of times an article is cited. They checked the citation frequency of each article for a certain year, in this case both 1985 and 2000, in the Journal of the American Society for Information Science and Technology (JASIST). The search for 2000 showed Google Scholar had 4.5 more citations than Web of Science and 3.9 more citations than Scopus. But searching 1985 Web of Science had 8.7 more citations than Google Scholar and Google Scholar had "just" 2.9 more citations than Scopus. The major shortcoming in this article is that they never analyzed the inconsistencies with Google Scholar citation Search. The citation count doesn't always work properly. Here's an example of a record that indicates Cited by 15 (other sources):

When clicking this link Cited by 15 you will find only 14 citations:

Here's another example of an article by P Jacso himself. Cited by 3 sources according to Google Scholar:

When clicking this link Cited by 3 you will find only 2 citations:

This search on semiconductors is an example from Jacso. In this reference it seems like the article is published 2006, but checking the source shows it's published 1990 and 2006 is the starting page of the article:

Jacso has also pointed out the flaws of duplication in his article "As we may search" in Current Science. Google Scholar works hard with the ability to cluster duplicate articles together. If you look at the preceding screenshot after the title you see the link group of 3>>. Clicking that link shows you 3 duplicates. Because Google Scholar indexes not just peer-reviewed journal articles, but preprint archives, conference papers, master thesis, webpublished materials etc you understand they have a hard problem to discover duplicates.

Here's an example. Searching sojka modeling drop size distributions gives as the first hit an article by Bainsky and Sojka with title "Modeling drop size distributions". That article should be Cited by 7 other sources according to Google Scholar.

By clicking Cited by 7 you find 7 hits but two of them are duplicates. View the two titles "Modeling Spray Impingement using Linear Stability Theories for Droplet Shattering". Though the first title has a link to group of 2>>.

Conclusion: Don't trust the Google Scholar citation counting without manually checking it for inconsistencies in terms of counting and clustering duplicates.

It's not hard to find inconsistencies and flaws in Google Scholar. Some of them follow below.
This search on semiconductors is an example from Peter Jacso Google Scholar and The Scientist 2005 (Published on university homesite as extra material).

In this reference it seems like the article is published 2006, but checking the source shows it's published 1990 and 2006 is the starting page of the article:

Searching advanced search limiting the date range to 1995-2006 returns 135,000 hits. But extending the date range to 1985-2006 returns just 131,000 hits. How come, Google Scholar?

Another flaw in Google Scholar is the OR Boolean operator. In this case the result for: dahlqvist OR dahlquist is 16.200 which means there should be 500 documents with both dahlqvist AND dahlquist, otherwise 16.700. But it's not.

This is a quotation from Peter Jasco "As we may search" in Current Science Vol. 89, No. 9. (10 November), pp. 1537-1547.:

"G-S is a free service, and for many who consider it to be a gift for the world it may be anathema to say any but good words of it. It is also to be emphasized that it is a joint gift by some publishers and/or their digital facilitators (the content part), and Google (the software and the service operation part). If ISI or Elsevier could have received such unfettered access to the publishers’ archives for harvesting their sites offering standard-compliant metadata, they could probably sell their services – if not for free – at a fraction of their current price. Building a multi-million record database incurs multi-million dollar investment just to subscribe to the journals, administer their processing, and record their standard bibliographic data, abstract, and descriptors, for about 1 million papers per year in the most recent period".

Of course, Google Scholar also has problems similar to Web of Science and Scopus when indexing author Rantapaa Dahlqvist S. When searching via advanced scholar search in author field, the following advanced search operators are returned: author:Rantapaa Dahlqvist author:s and 41 hits.

One of the 41 hits have both first name initials SB. The result also includes 7 hits on author variant Dahlqvist SR. A search on Dahlqvist SR shows on the contrary 28 hits on that variant. And here we find the article that was published in Lancet and was misspelled in Web of Science as Dahlwvist SR.

In contrast, a search on Rantapaa-Dahlqvist S without ää returns just 35 hits(!).

Trying to search misspelling Rantapaa-Dahlquist S returns no hits while same misspelling with ää Rantapää-Dahlquist S returns 2 hits which are not included in the Rantapaa Dahlqvist S search of 41 hits.

But searching Rantapää S returns 43 hits included misspelled rantapää-dahlquist.

But this is not all. Searching Rantapaa SB gives one more hit not included in the 43 hits or 41 hits mentioned before. Such a mess!

Let’s try some other authors. Searching author:sojka author:p also returns hits of P Jakubus Z Sojka. This means that all first name initials with P and all surnames called Sojka are searched with this syntax.

To get refined matching, use quotation marks like this: author:”p sojka”. But still the problem exists that you have both pa and pe sojka. To exclude Paul E Sojka you could write: author:”p sojka” –author:”pe sojka”, but still you can’t be sure that all p Sojka records include just PA Sojka. And to find records by PA Sojka you can’t add it like an OR-statement. Didn’t work for me. You need to do a separate search: author:”PA Sojka”

And it’s not possible to a make limited search to address. But, as you may already have learnt from previous postings, author address search in Web of Science and Scopus is too inconsistent to suggest as a valuable method for refining your author search.
It is also worth noting that if you don’t restrict an author search to the author field in advanced search or with advanced search operators you will get hits where the author name exists in the fulltext of the articles which Google Scholar indexes parts of.

Conclusion: The same problems with author search as in Web of Science and Scopus exists in Google Scholar. And as I said in a previous posting, a proper publication list from the author is the best way to be sure to find every important article.

It's hard to make an easy and still deeply and thorough evaluation of subject coverage in Scopus, Web of Science and Google Scholar due to a lot of reasons. Especially because the databases in question do not use established thesauri. Though I made a small comparison between these multidisciplinary databases and PubMed.

I chose three MeSH terms (two of them with subheadings included) with three words included. I limited my search to 1996, mainly because Scopus subject coverage before 1996 is selective. The MeSH-terms were:

Hormone replacement therapy
Antifreeze proteins toxicity
Neonatal screening ethics

Result from PubMed searching MeSH database:

Result from Scopus searching field keywords:

Result if broading the search to title, abstracts and keywords.

Result from Web of Science when searching Topic in General search which include title, abstracts and Keywords (author keywords and keywords plus):

As for results in Google Scholar they are more hard to evaluate, because Google Scholar indexes significant parts of the fulltext. It's possible to limit to title search but not abstracts for example. A lot of the material Google Scholar indexes is retrieved from the open web and other material is Journal articles references (and fulltext) from publishers. Google Scholar has not integrated any thesauri for the article references, however. Instead they have 7 subject areas available for limiting in advanced search. As viewed in this screenshot one of the 7 subject areas is Medicine, Pharmacology and Veterinary Medicine. I made a limit to that subject area and timespan 1996-.

2310 hits are definitely more that the others but as you see the second hit is definitely of high relevance but the others have indexed the word ethics in the fulltext where the word ethics is part of a ethics committee and not necessarily relevant.

Screen shot of search on antifreeze proteins toxicity shows 60 hits:

Not all of these hits are relevant and some are hits from books.

Screen shot of search on hormone replacement therapy shows 26.200 hits:

Conclusion: It's not recommendable to use Web of Science, Scopus or Google Scholar when doing exhausitve, specific searches when all possible important records of current science have to be found. This is due to the fact that thesauri and controlled vocabulary are not integrated at all or not properly.

Broadening a subject search in Scopus from searching Keywords to searching Title, abstracts and keywords gives a higher recall but not in all cases relevant records. To broaden a search both Scopus and Google Scholar is recommended but not Web of Science which indexes less material from 1996.

Next Page »