Information on the working of search engines

Any search engine will work in a specific order and that is, crawling, deep crawling, depth-first search or DFS, Fresh Crawling Breadth-first search or BFS, Indexing and searching.

Search engines retrieve a lot of information about different web pages from the web and store it. The web crawler or a spider will do this retrieving. It is an automatic web browser and it follows all links that it can see.

Then it analyses the page content for indexing the web page. It uses the index database in order to store the data about these web pages so that it can be use for any query later on. Search engines such as Google will store either a part or the entire source page besides any information of the web pages. On the other hand, some search engines will store every information about the page.

The source page or cached page will have the actual text for search because it was originally indexed. Therefore, it can be quite useful when the current page content is updated and any old search terms are not on it any more. This problem is a milder type of linkrot.

However, Google is able to handle it and that makes it more user-friendly. This way the user is not surprised in the least when he finds that the search terms are there on the returned page. An increased relevance for searches makes such cached pages more useful apart from the fact that they contain information, which is not available anywhere else anymore.

Now, when some user begins a search on a search engine, they will type in the keywords. The engine will now look up in the index to provide a list of the web pages that best match the criteria of the search. The search results will also contain a brief summary about the title of the document including certain portions of its text.

In order to specify any search query more, most of the search engines use Boolean terms OR, AND, NOT. Proximity search is an advanced aspect and it allows the user to define even the distance between the keywords.

The effectiveness or usefulness of any search engine will definitely depend upon the relevance of its search results. There might be several thousands and millions of pages on the web that might contain a specific word. However, not all of them are relevant to the search.

Hence, the search engine employs various methods for ranking their results so that they can list the best page on the top. now, how a search engine gives these ranks is highly subjective and varies from one search engine to the other.

Besides, these methods also keep on changing with time as well as innovations. Google, MSN and Yahoo are the current top search engines.