Googlebot is the search bot software used by Google, which collects documents from the web to build a searchable index for the Google search engine.
If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt file, or by adding the meta tag
to the web page. Googlebot requests to Web servers are identifiable by a user-agent string containing "Googlebot" and a host address containing "googlebot.com".
A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides "Webmaster Tools" that allow website owners to throttle the crawl rate.
Other articles related to "googlebot":
... Google uses an automated program called the "Googlebot." This program crawls the internet looking for new sites to include in its index ... Once a site is found the Googlebot creates a "cached" version of the site ... version, the user can view a "snapshot" of the page as it appeared at the time the Googlebot found the site ...