Creating a web site isn’t sufficient. Getting listed in the major search engines is the critical aim of website owners to ensure a site gets visible in SERP for several keywords. This listing of a website and visibility of freshest content is large as a result of search engine spiders which crawl and index sites. Webmasters could control the way in which these robots parse websites by inserting instructions in a special file called robots.txt. Several pages of a WordPress website need not be indexed by the search engines.