INDEXED GETTING
The leading search engines such as Google and yahoo use crawlers to find pages for their algorithmic search results. Pages that are linked from other search engine indexed pages do not need to be submitted because they are found automatically. Some search engines, notably Yahoo, operate a paid submission service that guarantee crawling for either a set fee or cost per click. Such programs usually guarantee inclusion in the database, but do not guarantee specific ranking within the search results. Two major directories the yahoo directory and the open directory project both require manual submission and editorial review. Search engine crawlers may look at a number of different factors when crawling a site.

Preventing crawling
To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robot.txt file in the root directory. Google warned web masters that they should prevent indexing of internal search results because those results are considered search spam.

Categories: SEO(Search Engine Optimization)