The Google Crawler is Fast
It took the Google crawler all of 15 minutes to index my last post about PowerEdge Ubuntu RAID monitoring. That is utterly ridiculous. It boggles my mind that they can so quickly index, compute PageRank, and cache everything in that little amount of time. Perhaps they don’t do all of that right away. Maybe they assume a fresh post has a high PageRank. Maybe they really do compute the PageRank on the fly. Regardless, I’m totally amazed.
3 Comments so far
Leave a reply
15 minutes is fast, but Google’s job is made pretty easy by Wordpress’ use of XML-RPC pings.
I suppose what I’m most impressed with is Google’s ability to index and compute PageRank in such little amount of time. Regardless, thanks for the link, Sunil!
What exactly would they need to calculate in terms of PageRank?
If they know that your post is extremely fresh, couldn’t they operate under the assumption that no one else has linked to it yet (so initial PageRank is close to zero, or maybe some average of your older posts), and wait until they see incoming links in future posts from across the web?
Their indexing speed is nonetheless quite impressive.