Google Crawls 1 Trillion URLs

Post Excerpts form Google’s Official Blog
We’ve known it for a long time: the web is big. The first Google index in 1998 already had 26 million pages, and by 2000 the Google index reached the one billion mark. Over the last eight years, we’ve seen a lot of big numbers about how much content is really out there. Recently, even our search engineers stopped in awe about just how big the web is these days — when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!

The web has been growing – faster and even faster. Several thousand domains being registered every day, another several million content pages being posted everyday. How much of web URLs this would add up on the web?

The web world is vast. Information being loaded everyday on the web pages. But the question is about the legitimity of the information being flooded. How much of this content is worthwhile and unique?

Points of Concern
1. How much of the content on the web is duplicate?
Image a same news being published on several news websites. The same news being again republished by several other website owners (either through RSS feed or other mediums)

2. How legitimate the information is?
The advent of blogs has flooded the internet with new webpages as blog enable a normal non-technical person to write a web page. Anyone can write a web page now!

This entry was posted in Google and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *