Hardin MD Notes on the Net                 About Notes on the Net Hardin MD Subscribe to Hardin MD Notes

Searching for directory sites: Google falls for Yahoo! Hardin MD Notes, Sept 12, 2000 Eric Rumsey eric-rumsey@uiowa.edu >> Sep 14: Partial retraction of this article (Please read) <<   Pages in top 30 hits in Google searches   Last November, as reported in Google likes directory sites, I discovered that Google had the uncanny ability to sniff out high-quality, but little-known directory sites. As I discussed in that article, Google was able to do this because it ranks sites according to how many people make links to them, and smart people everywhere learn that directories are important, so they make many links to them. Now, however, I've found solid evidence that Google's new alliances, especially the pact with Yahoo! that was announced in June, seem to have compromised its ability to find directories. The remarkable thing in November was that Google not only found the high quality directories that we have found for the Hardin Meta Directory (Hardin MD), but often it even ranked them in the same relative order that we did. A prime example of this was Yahoo! -- While Yahoo! has moderately useful directory pages in all subjects, including the medical/health subjects in the the Hardin MD, it's basically a generalist tool, not taken seriously by people doing specialized research. So the the Hardin MD does include Yahoo! pages, but they're usually ranked well below other more comprehensive pages done by people who specialize in health and medicine. It was striking, back in November, that Google saw things much as we do -- Yahoo! pages were usually included in the first 100 hits, but they were ranked well below other directory sites, in most cases exactly the same ones that we also rank higher in the the Hardin MD. November - February: Google continues to be a directory hound Since November, I've continued to monitor the placement of directory sites in Google searches, doing the same thorough check every 3 months that I did originally in November. When I did the first check in November, pages from the Hardin MD itself were ranked highly, and, more importantly, the other sites that we list were also consistently ranked highly. One that we generally rank highly in the the Hardin MD, and that was ranked especially highly in Google searches, is MedWebPlus, an excellent and comprehensive site from Steve Foote and Andy Kogelnik, y-DNA, Inc, Atlanta, Georgia. Pages from MedWebPlus were in the top 100 hits in all of the 21 searches that we did, and in 5 cases, they were in the top 15 hits. Yahoo! pages, by contrast, were in the top 100 hits only 14 times, and none were in the top 15 hits. The next time I did the check, in February, several of the directories that we list in the Hardin MD moved up in Google rankings relative to the placement of Yahoo! pages. Again, as in November, pages from the Hardin MD and MedWebPlus were in the top 100 hits consistently. Yahoo! fell to only having 12 pages in the top 100 Google hits. March: Yahoo! begins its move In March, we noticed in our regular subject checking for the Hardin MD that Yahoo! pages seemed to be climbing in the rankings, so I did a smaller version of the survey I wasn't scheduled to do again until May, and I found that indeed, there was a distinct move up for Yahoo!. For the March mini-check, I looked at a subset of 8 subject searches from the thorough-check group of 21; I checked pages from the Hardin MD, MedWebPlus, and Yahoo!. In November and February, the average ranking of these sites in the 8 searches was similar. But in March things changed -- While the Hardin MD pages kept about the same average (10th), MedWebPlus dropped from about 20th to 35th, and Yahoo!'s average rank rose from about 60th to 26th. At the next thorough check in May, Yahoo! continued its rise in the rankings. For the first time, it now ranked in the top 100 hits in all of the 21 searches. In 13 cases it ranked in the top 15; in February, it had none in the top 15. August: Yahoo! -- Yes! Other directories -- No In August, Yahoo! rose dramatically again - It's average ranking had now climbed to 9th; in all but one search it was in the top 15 hits. Hardin MD pages were still in the top 100 hits in most cases, but in the top 15 hits only 3 times. The real shocker in August: MedWebPlus, the prime quality directory site formerly ranked near the top by Google, is not in the top 100 hits in a single case -- Gone without a trace! Without a trace indeed -- To see just how far MedWebPlus had fallen, I looked at the first 500 hits for one subject in which it had consistently been in the top 10 (rheumatology), and MedWebPlus was not even in the top 500 hits! (Nor was SciCentral, another high-quality directory site in the Hardin MD, which has consistently been in Google's top 100.) Beyond the sudden rise of the main US page for Yahoo!, which is what's reported here, since it's always the top Yahoo! page, equivalent pages from non-US versions of Yahoo! have also begun to appear in the top 100 Google hits, beginning in May -- In several cases in August, I found 7 separate hits for the same subject pages from non-US versions of Yahoo!, including Singapore, Hong Kong, India, Asia, United Kingdom, Canada, and Australia. Google losing its nose for directories So, the scenario becomes clear -- Before March Google did a superb job of finding high-quality directory sites. But in March (no doubt, as Google and Yahoo! negotiated the alliance that was announced in June), Google's directory-finding ability began to suffer, and it's been in steady decline ever since. Now, with the latest report from August showing that Google is not even able to find one of the cornerstones of medical directory sites, its directory-finding ability seems to be in free fall. The alliance with Yahoo!, of course, is not the only recent agreement made by Google with a large directory provider -- There's also the partnering with Open Directory, announced in March, to produce the Google Directory. Have I found that Open Directory and Google Directory pages have risen in Google rankings? Yes, they have. But their rise is of less concern to me than Yahoo!'s rise, for two reasons -- First, I consider Open Directory, and especially Google Directory, to be more useful tools than Yahoo!, which might conceivably be linked on the pages of subject specialists, and might therefore legitimately be found by Google's PageRank searching. Secondly, the rise of Open Directory pages has been more gradual than Yahoo!, another indication that it might actually be a result of Google's link analysis technology. The real concern, then, is Yahoo! -- The rise of Yahoo!'s pages in Google's rankings could not conceivably have occurred legitimately, as a result of Google's PageRank searching -- Anyone with experience in the realm of medical/health directories will realize that subject specialists who make web pages don't often make links to Yahoo!. And the abrupt way in which Yahoo! pages rose in Google's rankings, surreptitiously, just a few months before the Google-Yahoo! alliance was announced, leaves little doubt that Yahoo! pages have been marked to receive special attention in Google searches. Falling for Yahoo! The propping up of Yahoo! is bad enough, but the really unfortunate thing is that not only has Yahoo! been pushed up, but other directories -- exactly the ones that Google used to be so exceptional in finding -- have been dropped down, or even disappeared. That the people at Google have made a conscious policy decision to raise Yahoo! in the rankings seems beyond doubt. The question that remains, that may never be answered, then, is -- What's the connection between Yahoo!'s rise and the fall of other directories? Our reports from November through February show clearly that Google has the ability to target directory sites. So has this ability now been employed to lower the rankings of other directories in Google searches, to draw more attention to Yahoo!? Or could it be that in embracing Yahoo!, Google has made a basic change in their search algorithm? -- That their link analysis system formerly keyed on directory sites as "guides to the Internet," and that directories will now be de-emphasized? And what might be the consequences of that for Google searching in general? We'll be continuing to follow Google's ability to find directories, and we've recently started tracking other search engines as well, to see if they're catching on. So far, even though its showing distinct signs of decline, Google is still ahead of the pack. But other search engines have begun making claims that they're adopting Google-like link analysis methods, and the "hunt for directories" may soon heat up. Google | Hardin MD | MedWebPlus : Rheumatology For more: Yahoo! Selects Google as its Default Search Engine Provider [Google press release, June 26, 2000] Google Extends Award-Winning Search Service with Addition of Netscape's Open Directory Project [Google press release, March 15, 2000] Hardin Library for the Health Sciences, University of Iowa Please send comments to hardin-webmaster@uiowa.edu The URL for this page is http://hardinmd.lib.uiowa.edu Last updated Last updated Thursday, Dec 02, 2010