Information management >

Search Engines

Search engines are not all alike. Apart from searching different sites, they also use different procedures for searching. Although it's a good idea to have one favorite search engine (on the same principle that you're faster finding your groceries in a familiar supermarket), it's better not to restrict yourself to only one.

For a start, two types of search resources are commonly called "search engines" although the term is only accurate for one type.

Sites such as the very popular Yahoo are actually directories, where websites are organised into hierarchical categories by humans. This makes them good for searches such as "I wonder what's available on financial planning" (the Yahoo homepage would tell you there's a section on Finance; go to Finance and there's Financial Planning). It's good for browsing categories (when you're looking at Finance, you might find sub-categories that you hadn't thought of - e.g., I was looking for Personal Finance sites and saw the sub-category Socially Responsible Investing, which I hadn't thought of looking for, although it's a subject that interests me).

The downside to search directories is that because each site has to be reviewed and categorised by a person, only a fraction of available websites are represented. For an exhaustive search you need a search engine.

A search engine (properly called) is compiled by computer programs called robots or spiders or crawlers (because they creep through the Web collecting information). AltaVista, HotBot, and Lycos are all search engines.

No search engine however covers more than about 16% (according to a recent study by NEC Research Institute).

Different search engines program their robots to search for different information. They also process your queries in different ways.

For example,

• Lycos displays sites by popularity.
• Google ranks sites by the number of links the site has, and the importance of those linked sites (i.e., the number of links they have). Results do well in terms of relevance.
• Excite ranks sites according to the keywords on the page - this can result in a completely unimportant page (e.g., with copyright information) being ranked quite highly.

Because of these sorts of differences, you should choose your search tool partly by the sort of query you have. Thus, if you're looking for a specific site, you're better going for a search engine rather than a directory. Google is considered good for broad searches. Excite is said to be better for general questions. Hotbot is considered to be better for specific searches.

Selective directories are rightly becoming more popular. One good one is the Argus Clearinghouse.

The Argus Clearinghouse is, in their own words, "intended to be a resource that brings together finding aids for students, researchers, educators, and others interested in locating authoritative information on the Internet". It's advantage is that the sites it offers have been reviewed and rated. The rating is reviewed every year. The downside is the relatively small number of links it has. But as a starting point to find a "good" information site, this is a great place to begin. The Argus also has a useful bookmark page with links to "automated" search tools. I particularly like the way it's grouped the links according to whether they're for searching for general information, software, people, or communities. Look under Internet Search Center > All search tools.

For more detailed information about searching, check out these sites:

A tutorial from Berkeley University library

CNet's Ultimate Guide to search

About's About searching (no, I don't have a stutter. About, subtitled the Human Internet, is an information resource that uses people known as Guides to select sites for particular topics as well as develop original content.)