Good results are most important for search. Most valuable raw material is a well filled index.
Of course in the long run the whole internet should be indexed. In the mean time we may improve search result also with a smaller index, if the crawler is exactly indexing the pages, the users are looking for. In this way for the same index size we may improve its efficiency.

This is exactly what our new active, community directed crawler is intended for. Additional to crawling visited pages, FAROO is now able to crawl autonomously.  Crawler start points are derived from visited pages and searches of the FAROO users. If a search returns only few or no results, pages are crawled in real-time and included in the results of that search. While searching the community directed crawler increases the index exactly there where it’s needed. If there are missing results, gaps are instantly closed.

Active crawling increases the index size at faster pace and overcomes the chicken egg problem when crawling only visited pages with relatively few users. By active crawling also passive peers may contribute. Increasing the index becomes independent from browsing activity, in this way also pages get indexed which nobody from the current FAROO community visited before.

The improved efficiency and speed of crawling and indexing will provide you with richer results every day.

This entry was posted on Tuesday, January 8th, 2008 at 11:55 am and is filed under FAROO, Peer to peer. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.