USC
About this Article
Written by: Reid Hirata
Written on: November 7th, 2003
Tags: computer science
Thumbnail by: Wikimedia Commons
About the Author
In Fall 2003 Reid was a junior majoring in computer engineering/computer science at the University of Southern California and hails from the island of Oahu. He enjoys applying web programming and entrepreneurship in the development of his company, Xaveon Security.
Also in this Issue
Base IsolationWritten by: Andrew Jacobs
Broadband over Power LinesWritten by: Philip Rendina
Doping in Sports: Blood Oxygenation EnhancementWritten by: Michelle Venables
Shedding Light on BlindnessWritten by: Andrew Wong
The Beauty of Science: New Technologies in Art RestorationWritten by: Sam Bagwell
Stay Connected

Volume VI Issue I > Search Engines: Guiding You in the Digital World
The daily life of an average American is depending more and more on the Internet. Information and entertainment sites abound, but many of the billions of web pages are yet unknown. Search engines are the pipeline through which users can quickly find the content they desire. The history of online search engines dates back to the very beginnings of the Internet. They have evolved to become complex tools that index and rank the enormous and highly dynamic environment of the World Wide Web. Much can be learned from investigating the way search engines rank sites and what can be done by a webmaster to rank highly in the major search engines.

Introduction

When we open up a web browser, what is the first page we visit? More often than not, it is our favorite search engine. Maybe we even set it as our default home page because we use it so frequently. But what engineering lies behind this simple exterior allowing us to access hundreds of millions of web pages with a few keystrokes and the click of a mouse? What complex processes run from the time we click "search" to the time we see the results page? A bit more than we might think.
Search engines bind the World Wide Web together for the average Joe Surfer, linking him to previously unknown and unreachable sites that contain the information he desires. Perhaps this is how you, the reader, have found Illumin. Yet there are billions of web pages still unindexed by any search engine - the "invisible web". Pages that require a subscription or registration, as well as pages generated dynamically based on user input, are off limits to the search engines that scan the Internet for new sites [1]. This article will introduce you to the inner workings of search engines, paint a picture of the search engine scene today, and offer some tips for better searching along the way.

A Brief History

Although the Internet was officially born in 1968 through ARPANET, online searching took its first steps in 1964 when the concept was still in incubation. Teams at MIT, Data Corporation, and IBM, among others, concurrently developed many features that are still in use in modern search engines. Most of the improvements came as browsing features and additional functionality and flexibility when viewing search results since the basic searching functions had been in place from ordinary text searches [2]. Many search engine companies came and went over the next 30 years as early search algorithms became obsolete in the rapidly changing World Wide Web environment. Those who managed to adapt, such as Lycos and Altavista, still survive today, but none share the success of newcomer Google (see Fig. 1).
Wikimedia Commons
Figure 1: Various Search Engines of Today.

The Process

Identifying documents and displaying them to the user is a fine art. Search engine users are often frustrated when they fail to find results that they expect. Undoubtedly the necessary websites exist somewhere, but finding one page in a few billion does not make for good odds. Understanding the way search engines work goes a long way towards narrowing down search results to the best candidates.